RAND 


Assessing  the  Usability  and 
Usefulness  of  the  CAHPS 
Decision  Helper,  Version 


DISTRIBUTION  STATEMENT  A 

Approved  for  Public  Release 
Distribution  Unlimited 


Mina  Kimmerling,  Mark  Spranca,  David 
Kanouse,  and  Sue  Phillips 

DRU-2330-AHRQ 

August  2001 

Prepared  for  the  Agency  for  Healthcare  Research  and  Quality 


RAND  Health 


The  RAND  iinrestricted  draft  series  is  intended  to  transmit 
preliminary  results  of  RAND  research.  Unrestricted  drafts 
have  not  been  formally  reviewed  or  edited.  The  views  and 
conclusions  expressed  are  tentative.  A  draft  should  not  be 
cited  or  quoted  without  permission  of  the  author,  unless  the 
preface  grants  such  permission. 

RAND  is  a  nonprofit  institution  that  helps  improve  policy  and  decisionmaking  through  research  and  analysis. 

RAND's  publications  and  drafts  do  not  necessarily  reflect  the  opinions  or  policies  of  its  research  sponsors. 


PREFACE 


Health  care  consiomers  find  themselves  in  a  difficult  position. 
Decisions  about  health  care  plans  are  increasingly  complex  and  plan 
coverage  is  increasingly  expensive.  Yet,  the  information  needed  to  make 
more  informed,  cost  effective  decisions  is  not  easily  available  or 
accessible  to  many  Americans.  In  other  purchasing  areas  consumers  may  be 
able  to  find  objective  sources  that  provide  information  on  specific 
products  or  consumer  reports,  yet  this  tends  not  to  be  the  case  when 
choosing  a  health  plan.  As  a  result,  the  Agency  for  Health  Care  Policy 
and  Research  (now  the  Agency  for  Healthcare  Research  and  Quality)  has 
funded  the  Consumer  Assessment  of  Health  Plans  Study  (CAHPS®)  ,  a  five- 
year  research  project  that  responds  to  consumer  needs  for  information 
about  health  care  quality.  The  agency  awarded  cooperative  agreements  to 
RAND,  Harvard  Medical  School  and  the  Research  Triangle  Institute  to 
develop  surveys  and  reporting  kits  as  well  as  to  provide  objective 
health  care  information  to  both  the  privately  and  publicly  insured 
consumers.  The  computer  based  Decision  Helper,  developed  by  RAND  in 
collaboration  with  Kleimann  Communication  group,  American  Institute  for 
Research,  and  Harvard  Medical  School,  provides  potential  enrollees  with 
summary  information  about  health  plans  based  on  consumer  reports  and 
ratings  of  care. 

This  document  summarizes  results  of  two  rounds  of  usability 
testing  of  the  Decision  Helper,  version  2.0.  Cognitive  interviewing 
methods  were  used  to  assess  the  computer-based  tool  in  terms  of  its 
navigation,  structure,  format,  design  and  content.  Our  findings  are  of 
interest  to  health  care  consumers,  distributors,  providers,  researchers, 
practitioners,  and  employers  who  are  attempting  to  provide  their 
employees  with  plan  options. 

Although  this  document  has  not  been  reviewed  or  formally  edited, 
it  may  be  cited  without  obtaining  the  permission  of  the  authors. 
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SUMMARY 


The  computer-based  Decision  Helper  is  a  tool  used  to  report  how 
consumers  enrolled  in  health  plans  assess  their  experiences  in  those 
plans.  This  information  can  be  used  by  potential  enrollees  to  inform 
their  own  plan  choices .  The  Decision  Helper  was  one  of  the  reporting 
products,  along  with  templates  for  printed  reports,  that  we  developed  as 
part  of  the  Consumer  Assessment  of  Health  Plans  Study  (CAHPS®)  . 

Consumers^  reactions  to  the  computer-based  tool  were  used  to  assess  and 
revise  its  architecture,  user  interface,  and  content. 

We  conducted  two  rounds  of  usability  testing  with  the  Decision 
Helper.  In  each  round,  a  total  of  11  Medicaid  and  privately  insured 
participants  were  interviewed.  In  a  two-hour  one-on-one  interview, 
participants  first  reviewed  the  Decision  Helper  site  independently  while 
giving  feedback  on  their  overall  impressions  of  the  site.  They  then 
answered  specific  questions  about  individual  web  screens  and  gave  their 
opinions  on  the  usability  and  usefulness  of  these  screens  and  the  site 
as  a  whole.  Participants  also  completed  a  written  questionnaire,  which 
elicited  their  reactions  to  the  site  and  gathered  personal  background 
information,  such  as  participant's  age,  gender,  and  prior  computer 
experience . 

Although  the  protocols  used  in  the  two  rounds  were  similar  (see 
Appendix  1  and  2),  we  used  results  from  the  first  round  to  identify 
problems  with  the  site,  revised  the  site,  and  then  used  the  second  round 
of  testing  to  evaluate  whether  the  revisions  were  successful.  Because 
use  of  the  Web  to  assist  health-care  decision-making  has  begun  only 
recently,  it  was  particularly  important  to  understand  participants' 
level  of  comfort  with  the  site's  format  and  with  the  Web  as  a  mediiom  for 
gathering  information. 

Usability  testing  was  determined  to  be  the  best  method  of 
assessment  because  site  users  may  identify  problems  that  are  not 
immediately  obvious  to  the  survey  designer  or  to  the  Web  developer. 
Usability  testing,  when  incorporating  cognitive  testing  techniques, 
consists  of  presenting  the  user  with  a  document  or  website  in  order  to 
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elicit  comments  and  concerns  from  the  users'  point  of  view.  This  type  of 
testing  has  been  shown  to  identify  problems  such  as  questions  that  are 
interpreted  inconsistently  or  not  as  intended,  respondent  difficulties 
in  understanding  structure,  navigation,  or  content,  redundancy  in 
content,  difficulties  in  structure  or  flow,  wording  or  terminology  that 
may  be  particularly  confusing,  or  problems  understanding  the 
informational  medium  (Harris-Kojetin  et .  al . ,  1999). 

Participants  in  both  rounds  of  testing  indicated  that  the  Decision 
Helper  was  easy  to  use  and  would  be  helpful  if  they  were  selecting  a 
health  plan.  However,  subjects  in  round  one  reported  several  key 
problems  with  site  format,  navigation  and  content.  Although  most 
subjects  said  they  found  the  CAHPS®  survey  results  valuable,  they  were 
surprised  and  frustrated  that  the  tool  did  not  contain  "essential" 
information  on  costs,  coverage  and  provider  availability.  Participants 
who  had  little  computer  experience  had  difficulty  using  the  site  because 
they  did  not  understand  how  the  site  was  structured  or  how  to  navigate 
through  it , 
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INTRODUCTION 


Choosing  a  health  plan  is  an  important  but  difficult  decision  for 
consumers.  When  making  a  health  plan  decision,  consumers  should  be  able 
to  compare  available  plans  in  terms  of  benefits,  costs,  availability  of 
providers,  health  care  delivery  and  quality.  Comparative  information  on 
plans  has  often  been  limited  to  basic  cost  and  benefit  information  while 
information  on  quality  has  been  difficult  to  obtain. 

In  an  effort  to  give  health  care  consumers  new  information  with 
which  to  compare  the  quality  of  health  plans,  the  Agency  for  Healthcare 
Research  and  Quality  (AHRQ)  funded  the  Consigner  Assessment  of  Health 
Plans  Study  (CAHPS®) .  The  goal  of  CAHPS®  was  to  provide  consumers  with 
reliable  and  valid  information  that  they  could  use  to  select  high- 
quality  plans  that  fit  their  needs  and  circumstances  {Crofton,  Liibalin, 
and  Darby,  1999) .  Begun  in  1995,  this  five-year  project  funded  three 
grantees  (Harvard  University,  RAND,  and  Research  Triangle  Institute)  and 
one  contractor  (West at)  to  develop  and  test  survey  and  reporting  tools 
to  help  consumers  assess  the  quality  of  their  health  plans.  The  products 
of  this  research  include  a  state-of  the  art  survey  to  elicit  reliable 
and  valid  information  from  consumers  about  their  experiences  with  health 
plans  and  reporting  tools  to  convey  the  survey  results  to  consumers 
choosing  a  health  plan.  The  CAHPS®  grantees  developed  reporting 
templates  in  both  print  and  computer  formats.  The  printed  report  is  a 
booklet  called  Compare  Your  Health  Plan  Choices.  The  computer  tool  is 
called  the  Decision  Helper^.  This  paper  reports  on  testing  of  the 
usability  and  usefulness  of  the  computer-based  health  plan  decision 
support  tool,  the  Decision  Helper. 


^  The  two  versions  of  the  CAHPS®  Decision  Helper  described  in  this  paper 
can  be  found  at  http://www.rand.org/health/cahps/old/dhv2/welcomel.htm 
(Round  1)  and  http://www.rand.org/health/cahps/dh/index.html  (Round  2). 
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PURPOSE  OF  TESTING 

From  laboratory  experiments,  we  have  learned  that  consumers  who 
utilize  the  computer  or  the  paper  CARPS®  reports  choose  different  plans 
from  those  who  do  not  (Spranca  et  al.,  2000),  although  the  effects  of 
CARPS®  reports  on  health  plan  choices  in  the  real  world  appear  to  be 
modest  at  best  (Farley  et  al.,  2001;  Farley  et  al,  in  press) .  Evidence 
from  filed  demonstrations  suggests  that  among  those  who  are  mailed 
printed  CARPS®  reports,  the  percentage  who  look  at  it  ranges  from  24%  to 
almost  90%  (Short  et  al . ,  2001).  Because  web-based  materials  are 
accessed  by  users  rather  than  distributed  to  them,  it  is  more  difficult 
to  determine  rates  of  use  for  computer-based  reports.  Nevertheless,  it 
is  clear  that  not  everyone  who  is  exposed  to  the  Decision  Relper  will 
use  it. 

In  an  effort  to  improve  the  likelihood  that  consumers  will  use  the 
Decision  Relper,  we  set  out  to  evaluate  the  usability  and  usefulness  of 
the  CARPS®  Decision  Helper.  We  felt  that  usability  is  a  necessary  but 
not  sufficient  characteristic  for  a  useful  site.  Usability  has  to  do 
with  such  things  as  whether  participants  are  able  to  easily  navigate 
through  the  site,  the  content  is  clear,  and  the  information  on  the  site 
is  well  organized.  For  the  site  to  be  truly  useful,  users  needed  to 
feel  that  the  Decision  Helper  aids  them  in  the  process  of  selecting  the 
best  health  plan.  To  assess  the  usability  of  the  Decision  Helper,  we 
tested  the  ease  of  site  navigation  and  the  comprehensibility  of 
information  on  the  site.  To  assess  its  overall  usefulness,  we  asked 
participants  how  valuable  they  thought  the  Decision  Relper  would  be  if 
they  were  to  switch  plans,  choose  a  new  plan  or  compare  their  current 
plans  to  the  health  plans  being  rated.  We  also  asked  users  to  assess 
usefulness  more  specifically  by  discussing  the  extent  to  which  each 
screen  enhanced  the  value  of  the  site  as  a  whole. 

As  the  CARPS®  Decision  Helper  is  designed  for  a  diverse  population 
of  privately  insured  individuals  and  Medicaid  decision-makers,  use  of 
the  tool  requires  a  basic  familiarity  with  a  web  browser. 

From  December  1  to  December  15,  1999,  11  participants  came  to  RAND 
to  view  and  comment  on  the  CARPS®  Decision  Helper.  After  their  comments 
had  been  analyzed  and  the  Decision  Helper  had  been  modified,  11 
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different  participants  viewed  and  coinmented  on  the  revised  website. 

This  report  addresses  the  findings  and  methodology  used  to  elicit  the  22 
participants'  comments,  concerns  and  suggestions  on  how  to  improve  the 
Decision  Helper's  structure,  navigational  cues,  and  content.  It  also 
describes  the  changes  we  made  as  a  result  of  the  usability  testing. 
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METHODS 


SAMPLE 

In  both  rounds  one  and  two  of  usability  testing,  the  22 
participants  varied  in  Medicaid  coverage,  age,  professional  status  and 
ethnicity  (See  Table  1) .  Ten  participants  were  Medicaid  recipients  and 
12  were  privately  insured. 


Table  1 :  Demographic  Characteristics  of  Medi-Cal  and  Privately  Insured 

Participants 


DEMOGRAPHIC  VARIABLES 

PERCENTAGE  OF 
MEDI-CAL 
PARTICIPANTS 
(N=10) 

PERCENTAGE  OF 
PRIVATELY 
INSURED 
PARTICIPANTS 
(N=12) 

Participants  aged  under  34 

40.0 

33.3 

Participants  aged  35~44 

20.0 

25.0 

Participants  aged  45  + 

40.0 

41.7 

Female 

80.0 

50.0 

Caucasian 

30.0 

66.7 

African-American 

50.0 

8.3 

Asian 

0.0 

16.7 

Hispanic 

20.0 

8.3 

With  at  least  4  years  of 
college  education 

20.0 

58.3 

"Excellent  or  Very  Good" 
self-reported  health 

70.0 

75.0 

Spends  greater  than  10 

(n=8) 

(n=9) 

hours/week  using  computer 

12.5 

88.9 

Participants  with  children 

70.0 

33.3 

Income  under  $35,000  Fiscal 
Year  1999 

100.0 

17.0 

PROCEDURE 

We  conducted  two  rounds  of  testing  to  assess  the  Decision  Helper' s 
perceived  usability  and  usefulness.  Round  one  consisted  of  11 
interviews.  After  making  revisions  based  on  the  results  of  round  one,  we 
conducted  a  second  round  of  testing  with  an  additional  11  eleven 
participants  to  assess  the  revised  Decision  Helper.  Participants  from 
rounds  one  and  two  of  testing  were  recruited  by  Focus  L.A.,  a  focus 
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group  recruiting  firm.  Participants  provided  oral  consent  to 
participate  in  the  study  at  the  time  of  recruitment.  A  single 
interviewer  conducted  each  two-hour  interview.  After  participants  had 
arrived  at  RAND,  the  interviewer  explained  that  the  Decision  Helper  was 
designed  to  help  users  choose  a  health  plan.  In  the  first  part  of  the 
interview,  participants  were  given  as  much  time  as  they  needed  (25 
minutes  on  average)  to  view  the  site;  they  did  not  have  to  look  at  every 
screen  or  follow  a  particular  order.  Participants  were  asked  to  give 
feedback  in  a  "think-aloud"  manner  as  they  reviewed  the  site.  After 
they  had  completed  their  review  of  the  site,  participants  were  asked  to 
give  their  overall  impressions,  describe  any  aspects  that  they  found 
confusing,  identify  features  they  thought  were  especially  effective, 
etc.  (see  Appendix  1  for  the  full  protocol)  .  The  participant  and 
interviewer  then  reviewed  the  individual  site  screens,  discussing  the 
detailed  features  of  each  screen  and  its  place  within  the  site. 
Participants  also  completed  a  written  questionnaire  that  elicited 
demographic  information  and  asked  their  opinions  on  the  site  as  a  whole 
and  on  the  individual  screens .  Although  the  questionnaires  were  the 
same  for  both  privately  insured  and  Medicaid  participants,  the 
demographic  portion  was  altered  slightly  to  fit  the  applicable 
population.  (See  sample  demographic  questionnaire  for  the  privately 
insured  in  Appendix  2) . 

In  order  to  assess  the  site's  usability  and  usefulness,  we  used  an 
iterative  approach  to  testing.  We  conducted  two  phases  of  one-on-one, 
interviewee-interviewer,  usability  testing.  This  method  allowed  us  to 
assess  reactions  to  the  Decision  Helper,  identify  problems  and  develop  a 
set  of  changes  designed  to  address  them,  implement  these  changes  to  the 
Decision  Helper,  and  then  test  the  effect  that  these  changes  had  on  the 
tool's  usability  and  usefulness.  It  also  allowed  us  to  determine 
whether  the  changes  actually  yielded  the  intended  improvements  and  to 
assess  whether  any  new  problems  were  introduced  as  unintended  side 
effects . 

In  conducting  the  usability  interviews,  we  utilized  two  main 
strategies:  retrospective  and  concurrent  assessment.  During  the 
unstructured  interviews  we  utilized  the  retrospective  approach.  The 
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interviewer  allowed  the  participant  to  review  the  site  in  full  and  then 
asked  for  overall  impressions  of  the  site  and  of  individual  screens. 
During  the  structured  interview  we  utilized  the  concurrent  strategy:  the 
user  viewed  the  website  while  answering  questions  about  each  screen, 
such  as  ("Where  do  you  think  this  link  could  take  you") ,  and  was 
encouraged  to  "think-aloud"  while  reviewing  particular  web  screens.  The 
information  elicited  through  the  retrospective  approach  has  been  shown 
to  be  useful  in  detecting  problems  with  overall  site  structure  and 
navigation.  The  concurrent  approach  is  particularly  useful  for 
examining  potential  problems  with  individual  screens  and  the  linkages 
between  screens.  For  example,  participants  were  not  only  asked  their 
impressions  of  each  screen,  but  were  also  asked  to  explain  the  meaning 
of  a  section  of  text  in  their  own  words .  Using  both  the  concurrent  and 
retrospective  techniques  of  assessment  provided  for  a  more  comprehensive 
review  of  the  site's  usability  than  would  use  of  either  strategy  on  its 
own. 


Structure  and  Content 

The  original  version  of  the  Decision  Helper  tested  in  round  one 
and  the  revised  version  tested  in  round  two  had  similar  site  structures . 
The  first  screen  was  a  "Splash"  screen  designed  to  attract  the  user's 
attention,  provide  a  short  summary  of  the  website,  and  motivate  the  user 
to  begin  using  the  site.  The  following  screens  were  grouped  into  four 
major  sections.  The  first  was  the  "Welcome"  section,  which  included  two 
or  three  screens  that  introduced  the  health  plan  choices  available,  the 
importance  of  making  a  good  choice,  and  provided  basic  guidance  about 
using  the  Decision  Helper.  The  "Types  of  Plans"  section  consisted  of  two 
screens  that  explained  the  basic  differences  between  the  types  of  plans 
the  user  was  to  choose  from  (i.e.,  HMD,  PPO,  FFS) .  The  "Survey  Results" 
section  consisted  of  four  or  five  primary  screens  describing  the 
importance  of  understanding  quality  of  care  in  making  a  health  plan 
decision,  and  more  than  25  secondary-level  screens  presenting  survey 
results  data.  The  survey  results  reported  users'  opinions  of  their 
health  plans  on  seven  topics:  Getting  Care  That  Is  Needed,  Getting  Care 
Without  Long  Waits,  How  Well  Doctors  Communicate,  How  People  Rated  Their 
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Health  Care,  Courtesy,  Respect  and  Helpfulness  of  Office  Staff,  Health 
Plan  Customer  Service,  and  How  People  Rated  Their  Health  Plan.  The 
survey  results  were  displayed  in  two  formats:  (1)  summary  tables 
comparing  health  plans  on  each  of  the  seven  topics  described  above, 
using  one,  two  or  three  stars  to  show  plans  that  were  worse  than  the 
survey  average,  about  the  same  as  the  survey  average,  and  above  the 
survey  average  for  all  plans,  respectively;  (2)  detail  tables,  showing 
the  questions  asked  for  each  topic  and  summarizing  the  overall  response 
distribution  in  bar  charts.  The  fourth  and  final  section  was  "Your 
Worksheet,"  which  included  three  to  four  screens  that  summarized  all  the 
information  presented  with  a  customized  worksheet,  and  presented  advice 
to  the  user  about  important  factors  to  consider  when  making  a  choice, 
such  as  doctors,  costs  and  benefits.  During  the  unstructured  interview, 
participants  were  free  to  explore  the  site  in  the  order  they  wished. 
During  the  structured  portion  of  the  interview,  the  interviewee  and 
interviewer  reviewed  the  site  in  the  order  listed  above. 

Both  the  original  and  revised  versions  of  the  Decision  Helper  had 
similar  layouts  and  navigational  aids.  In  both  cases,  the  layout 
consisted  of  a  primary  navigation  area  on  the  left  side  of  the  screen 
and  a  larger  content  area  on  the  right.  The  content  area  comprised 
text,  data  tables,  and  active  hypertext  "links"  (underlined  text)  that 
allowed  the  user  to  navigate  to  other  screens  and  to  obtain  related 
information  or  move  to  another  topic.  The  text,  tables,  and  hypertext 
links  were  written  in  HTML,  a  common  formatting  language  used  to  program 
websites.  The  data  tables  were  used  either  for  data  collection  or  for 
data  display.  The  primary  navigation  area  consisted  of  hypertext  links 
grouped  into  four  sets  of  navigation  links,  organized  according  to  the 
four  sections  described  above. 

Certain  aspects  of  the  primary  navigation  area  in  the  version 
tested  in  round  one  were  changed  in  round  two.  The  background  color  of 
the  navigation  area  in  round  one  matched  the  background  color  of  the 
main  content  area  (white  with  black  text,  for  maximum  readability) .  In 
round  two,  the  content  area  retained  the  white  background,  but  a  light 
tan  color  was  used  in  the  background  of  the  navigation  area,  to  help  the 
user  distinguish  between  words  that  were  to  be  read  carefully  for 
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content  and  those  that  were  simply  to  be  used  for  navigation.  The 
coloring  gave  the  main  text  prominence,  while  causing  the  navigation 
area  to  recede  into  the  background  instead  of  competing  for  the  user's 
attention.  The  primary  navigation  area  also  differed  between  round  one 
and  two  in  the  number  of  hypertext  links  presented  and  the  color  of  the 
links.  The  version  used  in  round  two  incorporated  improvements  to  the 
content,  which  resulted  in  fewer  navigation  links  overall  that  were 
better  organized  and  more  clearly  labeled.  In  round  one,  the  links  that 
the  user  had  not  yet  clicked  on  were  black,  while  those  that  the  user 
had  clicked  on  were  blue.  In  the  revised  version  of  the  site  all  links 
were  blue  and  link  color  was  not  dependent  on  whether  the  links  had  been 
utilized.  The  primary  navigation  tool,  the  Contents  Box,  was 
significantly  changed.  In  round  two,  the  Contents  Box  displayed  all  the 
primary  navigation  links  at  a  glance,  in  contrast  to  the  round  one 
version,  which  only  displayed  the  four  primary  section  links  along  with 
those  sub-links  for  the  section  that  the  user  was  currently  viewing. 

The  round  two  version  displayed  all  the  primary  navigation  links  at  once 
so  that  the  user  could  see  at  a  glance  how  many  total  screens  were 
available  and  see  a  screen's  location  within  the  website's  overall 
structure . 

Both  versions  incorporated  secondary  navigation  in  the  form  of  a 
"next /continue"  button,  to  allow  the  user  to  proceed  to  the  next  screen 
in  a  predetermined  sequence  (in  contrast  to  the  more  exploratory 
navigation  structure  available  in  the  aforementioned  primary  navigation 
bar,  which  allowed  users  to  browse  the  different  sections  of  the  site  in 
the  order  of  their  choosing) .  However,  the  two  versions  differed  in  the 
location  and  prominence  of  the  secondary  navigation.  The  version  in 
round  one  presented  the  "next"  button  as  a  large  graphical  button  in  the 
upper  left  portion  of  the  screen,  above  the  primary  navigation  area; 
however,  the  version  tested  in  round  two  included  a  hypertext  link 
labeled  "continue",  located  at  the  top  of  the  content  area  along  with 
the  main  body  of  the  text.  The  new  location  chosen  for  the  secondary 
navigation  was  designed  to  make  it  easier  for  the  user  to  move  forward 
to  the  next  screen.  Placing  the  link  within  the  content  area  made  it 
more  convenient  for  users  to  continue  through  the  site  when  they 
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finished  viewing  the  content  of  each  screen^  instead  of  requiring  users 
to  shift  their  view  to  the  left  side  of  the  screen  (away  from  the 
content)  each  time  they  wished  to  continue, 

MEASURES 

Four  types  of  data  were  collected  through  the  usability  tests: 
observational  data,  unstructured  feedback,  structured  comments  and 
^^itten  feedback.  The  observational  data  were  recorded  as  the  user 
freely  viewed  the  website.  Instructing  users  to  use  the  site  as  they 
would  at  home  or  in  another  unobservable  place,  the  interviewer  was  able 
to  see  what  screens  were  skipped,  which  screens  were  viewed  the  most, 
which  navigation  tools  were  used,  etc.  Because  the  user  was  allowed  to 
freely  search  the  site,  the  observation  part  of  the  interview  was 
representative  of  how  the  average  user  would  approach  and  navigate 
through  the  site.  During  the  unstructured  portion  of  the  interview,  the 
interviewer  asked  users  to  verbalize  their  overall  opinions  of  the  site. 

gave  both  negative  and  positive  feedback  on  site  structure, 
content,  navigation  techniques,  background,  font,  screen  color  and  any 
other  aspects  of  the  site  that  stood  out .  This  unstructured  portion  of 
the  interview  was  key  because  it  allowed  the  interviewer  to  compare 
respondents'  comments.  In  the  first  round  of  testing,  participants 
consistently  mentioned  problems  with  the  same  sections  of  the  site,  so 
that  the  areas  in  which  changes  were  needed  were  apparent. 

During  the  structured  section  of  the  interview,  the  interviewer 
and  the  participant  explored  the  site  in  a  more  orderly  fashion. 
Participants'  reactions  to  each  screen  were  gathered.  They  were  asked 
how  they  interpreted  each  screen  as  well  as  the  value  they  thought  each 
screen  added  to  the  site.  They  described  how  each  screen  fit  in  with 
the  rest  of  the  site,  how  easy  each  was  to  understand,  how  useful  each 
screen  was,  and  how  individual  screens  might  be  improved.  (See  Appendix 
2  for  specific  questions) . 

During  the  written  part  of  the  usability  tests,  participants  were 
asked  to  select  one  of  the  plans,  describe  why  they  selected  that  plan, 
what  influenced  their  decision  and  what  more  the  site  could  have  done  to 
aid  in  their  decision  making.  We  also  assessed  participants' 
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impressions  of  the  site's  usefulness  for  comparing  health  plans, 
navigation,  ease  of  understanding,  information  offered,  and  areas  in 
need  of  improvement.  (See  Appendix  1  and  2  for  sample  protocols  and 
questionnaires.  Although  the  questionnaire  was  utilized  for  all 
participants,  only  slight  changes  were  made  between  cohorts.  Therefore, 
only  one  of  the  four  questionnaires  that  were  administered  is  displayed 
in  the  Appendix . ) 
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OVERALL  RESULTS 


SITE  STRUCTURE  AND  NAVIGATION 

Site  structure  and  navigation  needed  major  Improvement, 

During  round  one  of  testing  it  became  apparent  that  site  structure 
and  navigation  needed  to  be  more  user  friendly.  Participants  had 
significant  difficulty  locating  on-screen  links,  and  a  number  of  users 
felt  lost  while  navigating  through  the  site.  "I  just  don't  know  where 
to  go  here,  I  can't  tell  where  you  want  me  to  go  next."  When  we  asked 
participants  who  were  viewing  the  final  website  screens  to  return  to  a 
specific  screen  in  the  middle  of  the  site,  four  of  eleven  could  not  do 
so.  "Okay,  so  you  are  asking  me  to  go  back  to  the  first  survey  that  I 
looked  at.  Well  I  just  don't  see  how  to  do  it,  maybe  I  could  go  back 
one  screen  at  a  time,  but  I  don't  see  another  way." 

Participants  used  both  the  Contents  Box  and  arrow  keys  for 
navigation,  yet  suggested  that  more  instruction  was  needed  on  site 
navigation.  "So  I  can  use  this  box  on  the  left  side  to  get  around  the 
site?  I  didn't  see  that,  usually  links  stand  out  more  clearly."  They 
also  felt  that  providing  a  clearer  structure  would  decrease  the 
likelihood  that  a  user  would  "lose  his  place"  on  the  site.  Because  the 
general  rather  than  specific  headings  of  the  site  were  listed  in  the 
Contents  Box,  users  were  also  unclear  which  sub-topic  screens  fit  with 
which  general  topic  screens.  As  one  interviewee  said,  "So  I  am  reviewing 
the  results  of  the  adult  survey,  well  I  guess  that  the  next  place  that 
I'm  supposed  to  go  is  to  the  'Worksheet'  screen."  An  unclear  structure 
and  navigation  path  led  participants  to  miss  significant  parts  of  the 
website.  For  example,  the  "Children's  Care"  and  "People  Who  Use  A  Lot 
of  Care"  titles  did  not  appear  in  the  Contents  Box  unless  the  user 
actually  visited  the  "Other  Survey  Results"  section,  when  the  contents 
for  that  section  expanded.  From  elsewhere  in  the  "Survey  Results" 
section,  the  heading  "Other  Survey  Results"  stood  alone.  As  a  result, 
four  of  the  eleven  participants  in  the  first  round  of  testing  overlooked 
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this  information,  not  realizing  that  it  was  there  or  assuming  that 
"Other  Survey  Results"  referred  to  additional  results  for  the  general 
adult  survey  population.  Participants  also  found  it  difficult  to 
distinguish  the  "Summary"  and  "Details"  screens  for  "Adults," 

"Children's  Care,"  and  "People  Who  Use  Lots  of  Care".  As  one 
participant  said,  "I  feel  like  I'm  seeing  the  exact  same  screen  over  and 
over  again,  I  just  saw  a  bunch  of  stars  rating  the  plans."  Another 
participant  commented,  "I'm  not  sure  whether  I'm  looking  at  the  adult  or 
child  survey,  there  is  no  way  for  me  to  tell  which  population  this 
survey  is  describing." 

Four  participants  initially  had  trouble  distinguishing  between  the 
"Summary  Chart"  and  the  "Review  the  Results"  screen.  As  one  participant 
noted,  "Didn't  I  just  see  this  screen?  It  is  reviewing  the  results  of 
the  plans  that  were  just  reviewed."  To  help  users  know  where  they  were 
on  the  site,  the  design  team  had  placed  a  red  arrow  within  the  Contents 
Box  that  pointed  to  the  header,  which  corresponded  with  the  current  web 
screen.  However,  many  participants  did  not  see  the  arrow  or  else  did  not 
realize  its  significance;  only  three  of  the  eleven  website  testing 
participants  found  it  useful. 

Several  changes  were  made  to  clarify  website  structure.  The  main 
change  was  the  expansion  of  the  Contents  Box,  By  having  the  four 
headings  ("Introduction,"  "Types  of  Plans,"  "Survey  Results,"  and  "Your 
Worksheet")  and  11  sub-headings  initially  visible  during  the  second 
round  of  testing,  rather  than  visible  only  after  one  of  the  main  topics 
is  selected  as  in  the  first  round  of  testing,  the  site  structure  was 
made  more  transparent  and  participants  had  an  easier  time  with 
navigation.  Additionally,  the  expanded  version  of  the  outline  used  a 
blue  ball  instead  of  the  red  arrow  to  maintain  a  place-keeping  icon  in 
the  Contents  Box.  As  a  result  of  these  changes,  participants  were  more 
likely  to  see  more  of  the  screens  on  the  site  during  the  unstructured 
observational  period  of  the  usability  testing.  However,  the  expanded 
Contents  Box  in  the  second  round  of  testing  did  not  help  users  navigate 
to  the  "Doctors",  Benefits",  and  "Costs"  links  on  the  "Your  Worksheet" 
screen.  Participants  were  not  able  to  see  that  these  3  titles 
represented  links  within  the  "Your  Worksheet"  screen  and  therefore  were 
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prone  to  skipping  these  screens.  Only  four  of  the  eleven  interviewees 
in  round  two  and  one  of  the  interviewees  in  round  one  saw  the  links  to 
these  screens  on  the  "Your  Worksheet"  screen. 

A  few  other  additions  and  changes  to  the  site  were  made  to  improve 
the  clarity  of  site  structure.  A  brief  description  of  how  to  use  the 
Contents  Box  and  the  "Continue"  key  (which  replaced  the  back  and  forward 
arrows)  was  placed  on  the  "Introduction"  screens.  A  screen  titled 
"What's  in  the  Decision  Helper?"  was  retitled  "Getting  Started,"  and 
instruction  on  use  of  the  continue  key  follow  a  brief  description  of  how 
the  site  was  structured.  In  the  Survey  Results  section,  the  "Other 
Survey  Results"  sub-heading  was  replaced  by  two  more  specific  titles, 
"Results  for  Children"  and  "Results  for  3+  Visit  Users"  to  clarify 
exactly  what  population  was  being  represented  by  the  "Other  Survey 
Results".  Additionally,  a  general  category  topic  and  sub-category  topic 
heading  was  added  to  each  screen  to  help  users  identify  where  they  were 
on  the  site.  For  example,  on  the  screen  explaining  the  differences 
between  HMOs  and  PPOs,  the  title  of  this  screen  includes  the  general 
heading,  "Types  of  Plans,"  as  well  as  the  siib-heading,  "HMOs  and  PPOs". 

Because  participants  in  the  first  round  of  testing  had  some 
trouble  personalizing  the  "Review  the  Results"  and  "Your  Worksheet" 
screens,  two  instructional  pop-up  screens  were  added  to  the  site  in 
order  to  explain  to  users  how  and  why  they  should  enter  infoinnation  on 
these  two  personalizable  web  screens.  When  the  22  participants  were 
asked  whether  it  was  easy  for  them  to  find  the  information  they  were 
looking  for  on  the  site,  four  participants  in  the  first  round  of  testing 
and  six  participants  in  the  second  round  said  that  it  was  "very  easy," 
six  participants  in  the  first  round  of  testing  and  five  in  the  second 
round  stated  that  it  was  "somewhat  easy,"  and  one  participant  in  the 
first  round  of  testing  said  it  was  "somewhat  hard"  (See  Figure  1) . 
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■  Round  I  B  Round  11 


Figure  1 :  Ease  of  Understanding  Site  Information 


USABILITY 

We  identified  four  general  areas  in  which  there  were  problems  with 
site  usability:  the  site  structure  needed  to  be  made  clearer, 
navigational  cues  needed  to  be  more  apparent  and  user  friendly,  certain 
headers  needed  to  be  made  more  descriptive,  and  various  aspects  of  user 
interface  needed  improvement.  We  discuss  each  of  these  areas  in  turn. 

Those  with  little  web/ computer  experience  had  difficulty  understanding 
site  structure  and  navigation. 

A  significant  finding  was  that  participants  who  had  little 
experience  with  computers  or  the  Web  had  difficulty  understanding  how 
the  site  was  structured  and  how  to  navigate  through  it.  This  was  true 
even  though  participants  received  instruction  on  how  to  locate  website 
links,  basic  navigation  (using  the  Contents  Box  or  the  arrow  icons),  and 
the  interactive  nature  of  the  web  screens  (e.g.,  information  entered  on 
the  "Review  the  Results"  screen  would  be  integrated  onto  the  "Your 
Worksheet"  screen) .  Participants  who  did  not  have  previous  exposure  to 
computers  and/ or  the  web  did  not  feel  comfortable  relying  on  site 
information.  As  one  woman  said  "I  just  don't  get  this.  I'd  rather  just 
look  over  the  booklet  that  the  health  company  gives  me."  For  these 
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users,  being  uncomfortable  with  the  computer  as  a  medium  for  health  plan 
information  led  to  difficulties  in  understanding  how  to  use  the  site. 

One  participant  insisted  that  "there  is  just  no  way  I'm  going  to  use 
this  thing  to  decide  who  my  baby  sees  as  their  doctor."  Like  this 
woman,  some  participants  had  not  been  exposed  to  the  Web  and  did  not 
understand  or  trust  it  as  a  way  to  make  "real  life"  decisions.  However, 
at  least  one  study  has  shown  that  consumers  who  have  had  previous  Web 
exposure  don't  always  trust  information  coming  from  the  Web  {Berry  et 
al.,  2000) . 

For  those  lacking  Web  experience,  problems  with  the  site  began 
immediately;  two  participants  had  difficulty  identifying  the  start 
button  on  the  "Splash"  screen.  In  contrast,  participants  who  had 
previous  computer  and  Web  exposure  (assessed  by  self-reported  use  of  at 
least  10  hours  of  computer  use  per  week)  reported  little  trouble  with 
overall  website  use  and  navigation.  Because  they  were  comfortable  with 
techniques  of  browser  use,  including  scrolling,  recognizing  active  links 
and  basic  site  navigation,  they  were  able  to  get  through  the  site 
faster.  Experienced  users  also  seemed  more  comfortable  than  less 
experienced  users  with  using  computer-based  information  to  choose  a 
health  plan.  Perceived  comprehensiveness  of  site  instruction, 
navigation,  content  and  site  personalizability  were  dependent  on 
previous  Web  experience.  For  instance,  those  users  who  had  more 
extensive  Web  experience  were  more  likely  to  see  that  information  from 
the  "Review  the  Results"  was  carried  over  to  the  "Your  Worksheet" 
results.  Perhaps  because  experienced  users  knew  what  to  look  for  on  a 
site,  figuring  out  that  the  Contents  Box  was  a  site  outline  and  that 
blue  underlined  text  signified  an  active  link  was  easier  for  them. 

The  Contents  Box,  interactive  web  screens  and  other  navigational  tools 
needed  to  be  made  more  apparent  and  more  user  friendly. 

A  problem  noted  by  every  participant  in  the  first  round  of 
testing,  including  those  with  and  those  without  previous  computer 
experience,  was  the  difficulty  of  detecting  on-screen  links.  Of  the 
eleven  participants  in  the  first  round,  three  failed  to  realize  that  the 
Contents  Box  could  be  used  as  a  navigation  tool.  Ten  of  the  eleven 
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missed  the  "What  Do  the  Bar  Graphs  Tell  You"  and  the  "What  Do  the  Stars 
Tell  You"  screens.  Participants  also  failed  to  notice  the  link  between 
the  "Review  the  Results"  screen  and  the  "Worksheet"  screen,  the  links  to 
the  "Details"  and  "Summary"  screens  on  the  "Survey  Topics"  screen  and 
links  to  the  "Doctors,"  "Benefits,"  and  "Costs"  screens  on  the  "Your 
Worksheet"  screen.  As  one  first  round  participant  stated,  "I  don't  see 
how  I  can  get  from  one  survey  to  the  next  without  going  to  every  screen. 
There  is  no  way  to  jump  around".  Another  first  round  participant 
commented  on  difficulty  of  detecting  on-screen  links:  "You  see  the 
problem  is  that  the  text  is  all  the  same  color.  So  I  couldn't  see  that 
you  could  link  to  the  graphs  from  the  "Survey  Topics"  screen.  You  guys 
need  to  make  the  box  on  the  left  hand  side  (Contents  Box)  more  visible 
as  a  link." 

Additional  suggestions  for  improving  link  visibility  included: 
setting  the  color  apart  from  the  rest  of  the  text,  having  the  links 
flash,  including  a  descriptive  screen  in  the  introduction  that  discusses 
how  to  identify  links,  and  increasing  the  size  and  dimensions  of  the 
links .  To  address  the  problems  identified  by  the  first— round 
participants,  we  changed  all  of  the  link  colors  from  black  to  blue  with 
underlining  so  that  the  links  would  stand  out  from  the  black  text.  (On 
the  first  version  of  the  website,  un-clicked  links  were  black  in  text 
color.  Once  clicked  on,  the  link  text  became  blue).  We  also  lengthened 
the  Contents  Box  to  fill  the  left  side  of  the  screen.  During  the  first 
round  of  testing,  the  Contents  Box  listed  only  four  main  headings,  from 
which  sub-headings  would  appear  when  one  of  the  main  headings  was 
selected.  As  revised  for  the  second  round,  the  Contents  Box,  which 
listed  the  four  main  headings  with  11  links  to  sub— topics,  was  much  more 
prominently  displayed  on  the  screen. 

After  we  had  made  these  changes,  participants  found  the  site 
easier  to  navigate  and  the  site  information  easier  to  understand.  "It's 
pretty  clear  how  you  can  move  around  the  site.  By  clicking  on,  for 
example,  one  of  the  survey  topics,  I  can  see  how  the  topic  was  rated." 

As  the  links  on  the  "Review  the  Results"  screen  were  more  visible  and 
the  pop-up  boxes  were  added,  participants  were  more  likely  to  identify 
the  interactive  nature  of  the  site.  In  explaining  how  different  site 
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information  was  linked  by  different  screens  one  participant  said,  "After 
entering  my  opinions  on  each  of  the  plans  on  the  "Review  the  Results" 
screen,  the  information  is  transferred  over  to  the  "Worksheet"  screen." 
Few  round  one  participants  had  this  understanding  of  how  the  information 
on  one  screen  transferred  to  another.  Overall,  participants  had  a 
better  understanding  of  the  site  when  navigational  tools  were  made  more 
visible  and  were  better  explained  (see  Figure  2) .  As  a  result  of  these 
and  other  changes,  participants  in  the  second  round  of  usability  testing 
rated  the  website  more  highly  than  did  those  in  the  first  round. 


Figure  2 :  Ease  of  Website  Navigation 


Though  site  language  was  clear  and  comprehensive^  headers,  links,  and 
terms  needed  to  be  more  descriptive . 

Participants  in  the  first  round  of  testing  felt  that  the  language 
used  on  the  site  was  clear  and  easy  to  understand.  This  finding  bears 
on  a  key  aspect  of  usability.  The  language  used  to  describe  the  nuts 
and  bolts  of  health  insurance  coverage  and  systems  of  care  delivery 
includes  many  terms  and  concepts  that  are  difficult  for  consumers  to 
understand.  Selecting  a  health  plan  is  difficult  in  part  because  of  the 
complex  language  used  by  health  plans  in  describing  their  coverage  and 
the  various  costs  (premiums,  deductibles,  co-pays,  etc.)  included  in 
health  care  expenses .  It  was  important  that  the  language  used  in  the 


Decision  Helper  be  consistent  with  terminology  people  would  find  in 
other  materials  yet  be  as  clear  as  possible  to  people  of  all  educational 
backgrounds.  As  one  participant  said  about  the  "Types  of  Plans"  screen, 
"This  is  good,  the  information  is  clear,  I  can  see  the  comparisons  that 
you  are  trying  to  make  between  HMOs  and  PPOs . " 

Nevertheless,  participants  in  both  rounds  of  testing  felt  that 
some  screen  headers  and  links  were  not  accurate.  For  example,  one 
participant  expected  that  the  "doctors"  link  on  the  worksheet  screen 
would  give  information  on  the  specific  doctors  in  each  plan,  when  in 
reality  it  simply  listed  key  questions  to  ask  when  selecting  a  personal 
doctor.  "See  this  isn't  right,  I  came  to  this  screen  because  I  wanted  to 
see  who  I  could  choose  as  a  doctor  and  whether  my  doctor  would  be  on 
this  list.  I  don't  think  that  this  title  really  explains  what  the  screen 
shows" . 

Eighteen  of  the  21  participants  felt  that  the  description  of  the 
"People  Who  Use  A  Lot  Of  Care"  subgroup  as  individuals  who  use  care 
three  or  more  times  a  year  was  misleading.  Participants  said  that  going 
to  the  doctor  at  least  three  times  a  year  was  the  norm,  but  because  of 
the  title  they  skipped  these  seemingly  irrelevant  screens.  As  one  woman 
said,  "A  normal  person  goes  to  the  doctor  at  least  three  times  per  year. 
I  don't  think  that  should  count  as  using  lots  of  care."  Sample  size 
considerations  had  dictated  the  selection  of  "three  or  more  visits"  as 
the  threshold  for  defining  a  group  that  used  higher  amounts  of  care;  if 
the  threshold  were  set  at  a  higher  number  of  visits,  too  few  survey 
respondents  would  be  considered  high  users  of  care  to  permit  meaningful 
comparisons  among  plans.  However,  "three  or  more  visits"  does  not 
correspond  to  consumers'  notions  of  what  constitutes  "using  lots  of 
care."  Based  on  participants'  reactions,  we  revised  the  label  for  the 
section  on  this  cohort  to  "Results  for  People  with  3+  Visits." 
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The  user  interface  needed  Improvement. 


In  the  first  round  of  usability  testing,  monitors  were  set  to 
display  the  "normal"  size  font  (96  dpi) .  Four  participants  complained 
that  the  text  was  difficult  to  read  and  that  too  much  of  the  background 
was  visible.  Therefore,  monitor  settings  were  modified  in  the  second 
round  to  display  a  larger  font  (120  dpi) .  As  a  result  of  an  increase  in 
font  size  and  screen  size  none  of  the  participants  in  the  second  round 
of  testing  complained  about  reading  site  text. 

In  the  first  round,  four  participants  mentioned  that  they  found 
the  site's  bright  background  distracting  and  suggested  that  the  color  be 
changed.  As  one  participant  said,  "I  think  that  the  green  is  really 
distracting,  I  can't  concentrate  on  the  text  because  of  the  background 
is  so  noisy."  Suggestions  on  how  to  improve  the  background  ran  the 
gamut  from  using  a  plain  white  background  to  introducing  a  "Millennium 
theme"  with  balloons  and  streamers.  Three  participants  in  the  first 
round  of  testing  mentioned  that  they  had  difficulty  reading  text  on 
certain  screens  in  the  "Welcome"  and  "Review  the  Results"  sections  where 
the  title  or  introductory  information  was  placed  on  light  blue  or  green 
background.  "You  need  to  get  rid  of  the  background  color  behind  the 
titles.  For  some  reason  it  made  me  skip  over  the  words  and  not  read  the 
screen's  instructions."  Another  participant  commented,  "The  colors  need 
to  be  changed...keep  the  text  in  its  own  area,  without  any  colors,  but  put 
more  in  other  areas  of  the  screen." 

To  address  this  readability  problem,  the  Decision  Helper  design 
team  changed  the  screen  layout,  the  background  colors,  and  the  text  box 
colors.  The  version  of  the  Decision  Helper  that  we  tested  in  round  one 
had  arrow  links  on  every  screen  that  would  take  the  user  to  the  previous 
screen  or  next  screen  and  a  button  link  that  would  take  them  to  the  home 
screen.  Based  on  the  first  round  of  testing  results,  the  design  team 
replaced  these  arrow  keys  with  a  "continue"  link  on  the  upper  right  of 
each  screen  that  would  lead  the  user  to  the  next  screen  and  modified  the 
Contents  Box  to  provide  a  more  detailed  listing  of  site  contents  with 
active  links  to  other  sections.  The  design  team  increased  the  size  of 
the  font  and  made  the  text  boxes  larger  so  that  less  of  the  background 
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was  visible.  The  design  team  also  changed  the  background  colors  to  more 
neutral  tones  and  made  the  screen  titles  more  noticeable  by  eliminating 
the  surrounding  background  color  and  making  the  title  font  brighter. 
Participants  in  round  two  did  respond  more  positively  to  the  revised 
color  scheme.  As  one  user  said,  "These  pastel  colors  are  kind  of 
unusual,  but  I  guess  that  they  make  sense.  With  more  dull  colors  it 
doesn't  hurt  your  eyes  to  stare  at  the  screen." 

Results  from  the  second  round  indicated  that  the  changes  made  to 
the  site  as  a  result  of  the  round  one  users'  comments  improved  users' 
experience  with  the  site.  Participants  still  missed  a  number  of  links, 
but  the  changes  in  link  colors,  the  increase  in  the  link  font  size,  and 
the  inclusion  of  additional  instructions  on  navigation  greatly  improved 
user  interface.  The  disadvantage  of  some  required  scrolling  due  to  the 
increased  font  size  appeared  to  be  more  than  compensated  by  having  a 
more  readable  font . 

USEFUIiNESS 

Participants  who  were  unfamiliar  with  computers  and  the  net  did  not  find 
the  site  useful. 

Not  surprisingly,  the  Decision  Helper  appeared  much  more  valuable 
to  those  who  had  computer  experience  than  to  those  who  did  not.  As  one 
woman  said,  "I  don't  really  trust  computers  and  would  just  prefer  to 
have  the  health  plan  booklet."  Another  participant  made  a  similar 
statement:  "It's  just  not  the  way  that  I  usually  go  about  choosing  a 
plan,  so  I  don't  know  if  I'd  really  select  a  plan  that  a  computer 
recommended."  Five  of  the  seven  participants  without  computer 
experience  expressed  this  sentiment . 

Participants  needed  additional  logistical  information  to  make  final 
health  plan  decisions . 

Participants  of  all  levels  of  computer  experience  also  felt  that 
the  site  would  be  more  useful  if  "objective  information"  such  as  costs 
and  lists  of  doctors  available  in  the  plan  were  included.  The  CAHPS® 
Decision  Helper  version  2 . 0  focuses  on  health  plan  survey  information 
and  important  issues  to  consider  when  selecting  a  health  plan. 
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Therefore,  we  tested  only  the  template  for  presenting  CARPS®  survey 
information  and  did  not  test  an  expanded  version  that  includes  costs, 
benefits,  or  other  information  about  plans. 

Sixteen  of  the  21  participants  found  the  site  to  be  useful  overall 
but  said  that  they  would  have  a  difficult  time  using  only  the 
information  found  on  the  site  to  make  a  health  plan  decision.  Almost  all 
of  the  interview  participants  in  both  rounds  one  and  two  of  testing  felt 
that  they  could  not  come  to  a  definitive  choice  on  the  health  plans 
listed.  Seventeen  stated  that  they  would  need  information  on  covered 
services,  16  would  want  information  on  costs  (such  as  premiums,  co¬ 
payments  and  medication  costs)  for  each  plan,  12  would  want  information 
on  doctors  available  in  each  plan,  eight  would  want  information  on 
doctor  quality  and  one  stated  that  she  would  like  information  on 
facility  locations  (Figure  3) .  As  Table  2  shows,  participants  felt  that 
more  information  on  types  of  plans,  survey  results  and  other  information 
would  enhance  the  site's  usefulness. 

Figure  3:  Additional  Information  Requested  by  Participants 


Services  Cost  Doctors  Rating  of  Facility 
Covered  Information  Included  Doctor  Location 

Quality 


Type  of  Information  Requested 
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Table  2:  Amount  of  Information  Desired  Before  Choosing  a  Plan 


Number  of  Participants 


Survey  Response 

Types  of 
Plans 

Survey 

Results 

Other 

Information 

Much  more 

6 

1 

4 

A  little  more 

8 

10 

9 

Right  amount 

6 

7 

6 

A  little  less 

0 

2 

1 

Much  less 

0 

0 

0 

During  both  rounds  of  testing,  participants  were  asked  to  rate  the 
site's  usefulness  in  three  broad  areas:  comparing  plans,  choosing  a  new 
plan  and  switching  plans.  For  each  area,  response  options  were  "very 
useful,"  "somewhat  useful,"  "slightly  useful"  and  "not  at  all  useful." 

In  judging  the  site's  usefulness  for  comparing  their  current  health  plan 
and  other  plans,  four  participants  in  the  first  round  and  eight  in  the 
second  round  rated  the  Decision  Helper  as  being  "very  useful,"  three  in 
the  first  round  and  one  in  the  second  round  rated  it  as  being  "somewhat 
useful,"  and  four  in  the  first  round  and  two  in  the  second  round  rated 
it  as  "not  at  all  useful"  (Figure  4) .  Seven  participants  in  round  one 
and  six  participants  in  round  two  found  the  site  "very  useful"  for 
choosing  a  plan.  Four  participants  in  round  one  and  five  in  round  two 
found  it  "somewhat  useful"  for  choosing  a  plan.  Two  participants  in 
round  one  and  five  participants  in  round  two  found  the  site  "Very 
Useful"  if  the  user  was  to  be  switching  .plans .  Nine  participants  in 
round  one  and  five  in  round  two  found  it  "Somewhat  Useful"  when 
switching  plans.  One  participant  in  round  two  found  it  "Not  at  All 
Useful"  for  switching  plans. 
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Figure  4:  Usefulness  of  Decision  Helper  in  Comparing  Your  Health  Plan 

to  Other  Plans 


To  assess  the  usefulness  of  each  screen,  we  had  participants  rate 
their  impressions  of  individual  screens  during  both  rounds  of  testing 
(Table  3) .  For  each  screen,  participants  were  asked  to  rate  usability 
on  the  same  four-point  scale  used  to  rate  the  Decision  Helper  as  a 
whole,  from  "very  useful"  to  "not  at  all  useful." 

As  Table  3  shows,  more  participants  found  the  "Types  of  Plans" 
screen  at  least  "somewhat  useful"  than  found  the  "Welcome"  screens  at 
least  "somewhat  useful".  Many  participants  stated  that  they  found  the 
"Types  of  Plans"  screen  the  most  useful,  because  it  provided  novel  and 
objective  information  about  health  plans.  The  information  about  HMOs  and 
PPOs  was  stated  simply  enough  that  all  participants  could  comprehend  it, 
yet  was  complex  enough  to  cover  the  main  differences  between  types  of 
plans . 
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Table  3: Perceived  Usefulness  of  Individual  Screens 


Not  at  All 

Useful 

Slightly 

Useful 

Somewhat 

Useful 

Very  Useful 

Welcome 

Screen 

1  (5%) 

3  (15%) 

4  (19%) 

13  (62%) 

Types  of 
Plans 

0  (0%) 

1  (5%) 

6  (27%) 

15  (68%) 

Survey 

Results 

0  (0%) 

3  (15%) 

5  (25%) 

12  (60%) 

Children' s 

Care 

2  (10%) 

2  (10%) 

5  (25%) 

11  (55%) 

People  Who 
Use  Lots  of 

Care 

2  (10%) 

4  (20%) 

7  (35%) 

7  (35%) 

Summary 

Charts 

1  (5%) 

3  (14%) 

6  (27%) 

12  (54%) 

Details 

Charts 

2  (9%) 

0  (0%) 

7  (32%) 

13  (59%) 

Review  the 

Results 

0  (0%) 

4  (20%) 

9  (43%) 

8  (38%) 

Your 

Worksheet 

2  (12%) 

3  (18%) 

6  (35%) 

6  (35%) 

Participants  found  the  "Suinmary"  and  "Details"  screens  as  well  as 
the  survey  description  screen  very  valuable.  As  one  woman  stated,  "you 
just  have  to  have  a  screen  like  this  (survey  description  screen)...  if  I 
don't  know  where  the  information  is  coming  from  and  who  is  being 
sampled,  the  actual  results  don't  mean  anything."  A  few  participants 
missed  the  information  on  the  population  being  sampled  (300  adults  and 
parents  in  Los  Angeles)  as  well  as  who  conducted  the  survey  (an  outside, 
independent  firm)  .  Of  note,  it  was  these  participants  who  stated  that 
they  did  not  trust  the  survey  results  and  would  not  use  the  information 
supplied  to  help  them  choose  a  plan.  One  participant  who  missed  this 
information  stated,  "I  don't  know  who  put  out  this  information,  it  could 
be  the  health  plans  or  the  employer  who  will  give  you  biased 
information.  If  I  knew  that  it  was  a  reputable  source  that  supplied 
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this  data,  then  maybe  I  would  use  the  information  to  make  a  choice. 

After  the  information  about  plan  sponsors  and  participants  was  brought 
to  the  interviewee' s  attention,  she  felt  more  comfortable  selecting  a 
health  care  plan  based  on  the  website  information. 

Overall,  participants  in  both  rounds  of  testing  found  the  Review 
Your  Results"  and  the  "Worksheet"  screens  to  be  the  least  useful.  These 
two  screens  had  the  most  revisions  done  as  a  result  of  feedback  from  the 
first  round  of  testing.  However,  even  after  revisions  to  these  screens 
were  made,  round  two  participants  felt  that  the  interactive  nature  of 
the  screens  was  not  clear,  and  they  were  frustrated  that  these  screens 
did  not  provide  objective  information  on  each  of  the  plans.  After  round 
one,  a  pop-up  window  was  added  for  the  "Review  the  Results  screen 
explaining  that  the  answers  the  user  provided  on  this  screen  would 
appear  later  on  the  "Worksheet."  Though  the  pop-up  window  made  clear  how 
to  input  information  onto  the  "Review  the  Results"  screen,  participants 
felt  that  the  screens  would  be  more  useful  if  they  could  input  personal 
information  or  were  provided  with  information  on  actual  doctors,  costs 
and  benefits. 

The  "Summary"  chart  that  presented  comparative  plan  information 
and  the  "Details"  charts  that  summarized  absolute  results  were  central 
to  the  Decision  Helper.  Therefore  it  was  essential  that  participants  be 
able  to  understand  and  use  them.  Participants  realized  that  the  "Detail 
charts  contained  more  information,  but  they  felt  that  it  took  more 
effort  to  extract  this  information  than  it  did  to  glean  information  from 
the  "Summary"  chart.  For  those  who  wanted  a  quick  comparison  between 
plans,  the  "Summary"  chart  was  seen  as  easier  to  access  and  comprehend. 
However,  many  participants  mistakenly  thought  that  the  stars  in  the 
"Summary"  chart  represented  direct  ratings  of  each  plan  rather  than 
simply  a  comparison  of  where  each  plan  stood  in  relation  to  the  survey 
average.  Because  participants  differed  in  their  preferred  modes  for  how 
the  information  should  be  presented,  there  is  justification  for 
pr0S6nting  the  results  in  both  star  chart  and  bar  chart  formats. 
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INDIVIDUAL  SCREEN  RESULTS 


In  this  section  we  describe  participants'  reactions  to  the 
individual  screens.  For  each  set  of  screens,  we  present  the  results  in 
the  following  order:  coinments  of  first  round  participants,  changes  made 
as  a  result  of  these  comments,  reactions  of  second  round  participants, 
and  summary  of  overall  screen  usability. 

SPLASH  AND  INTRODUCTORY  SCREENS 

During  the  structured  interview,  participants  reviewed  the  site 
screen  by  screen  with  the  interviewer.  Participants  began  with  the 
first  screen,  the  "Splash"  screen,  and  were  asked  their  impressions  of 
the  photograph.  During  the  first  round  of  testing,  we  asked  participants 
for  opinions  on  the  photograph  displayed  in  the  version  initially  tested 
(see  Appendix  3) .  We  expected  that  parents  especially  would  identify 
with  this  photograph,  which  showed  a  mother  and  daughter.  However,  the 
only  participants  who  strongly  identified  with  the  photograph  were  two 
African  American  mothers.  Rather  than  identifying  with  the  family  theme 
in  the  photograph,  these  participants  thought  that  African  Americans 
were  being  targeted  by  the  website,  which  increased  their  interest  in 
the  site's  contents.  Other  first-round  participants  commented  on  "the 
warm  sniile  of  the  people  on  the  front  screen,  the  uncluttered  nature  of 
the  screen  and  the  simplicity  of  the  screen."  One  of  the  study 
participants  who  did  not  like  the  "Splash"  screen  reported  that  she 
didn't  feel  that  "the  picture  of  a  happy,  healthy  looking  mother  and 
daughter  duo  should  be  the  introduction  to  a  site  on  health  care.  There 
should  be  pictures  of  everyone  who  uses  health  care,  both  sick  and 
healthy  people." 

Although  no  changes  were  made  to  the  "Splash"  screen  between 
rounds  of  testing,  each  participant  in  the  second  round  of  testing  was 
shown  the  original  "Splash"  screen  and  an  additional  14  possible  screen 
pictures  (see  Appendix  5) .  The  original  screen  displayed  a  woman 
holding  her  daughter  and  smiling.  Other  screens  included  pictures  of 
doctors  and  families  and  scenes  from  both  medical  and  everyday  contexts. 
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A  majority  of  the  second- round  participants  preferred  Photo  G-  a  photo 
of  a  woman,  her  child,  and  a  doctor  examining  the  child.  A  number  of 
participants  also  preferred  a  photograph  of  an  infant  being  examined  by 
a  doctor.  (Many  of  the  comments  reflected  a  preference  for  the  picture 
to  be  focused  on  health  services  rather  than  on  the  family,  because  site 
users  would  have  different  family  situations  but  would  all  identify  with 
a  medical  setting.) 

Participants  in  round  one  recognized  that  the  "Welcome"  screens 
outlined  the  site  structure  and  the  target  audience.  "The  description 
on  this  screen  makes  it  easy  to  compare  it  to  that  text  on  the  left  side 
of  the  screen  (Contents  Box) ."  As  identified  earlier,  some  participants 
felt  that  the  Contents  Box  needed  to  be  better  explained  as  the  user  was 
being  introduced  to  the  site.  "You  don't  know  where  to  go  from  here. 
There  needs  to  be  some  explanation  of  how  to  get  around  the  site.  I  can 
see  the  forward/back  keys,  but  I  don't  see  any  other  way  to  get  around." 
Participants  did  have  some  difficulty  identifying  the  links  on  the 
"Welcome"  screens .  They  confused  the  bold  black  text  with  links  and 
were  confused  when  clicking  on  this  text  did  not  elicit  a  computer 
response . 

As  a  result  of  participants'  comments  on  the  usability  of  the 
"Welcome"  screens,  the  revised  version  tested  in  round  two  included  two 
"Continue"  links  that  led  users  to  the  next  screen.  Additionally,  the 
round  two  "Welcome"  screens  were  more  comprehensive  in  describing  what 
information  could  be  found  on  the  site  and  re-iterated  the  fact  that  the 
survey  was  done  by  an  independent  research  organization.  However, 
reactions  to  the  revised  "Welcome"  screen  in  round  two  testing  revealed 
little  change  in  participant  responses. 

In  terms  of  the  overall  usability  of  the  "Splash"  and  "Welcome" 
screens,  participants  were  able  to  determine  how  to  move  throughout  the 
site.  "Splash"  screen  usability  was  simple — a  large  yellow  arrow  with 
the  word  START  under  it,  indicating  where  participants  would  move  to 
next.  Participants  found  the  "Welcome"  screens  to  be  usable  as  an 
introduction  to  the  site,  but  even  after  more  explanatory  text  was 
added,  participants  commented  that  more  text  on  how  to  navigate  and  on 
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the  interactive  nature  of  the  site  and  the  various  site  sections  would 
have  been  useful. 

TYPES  OF  PLANS 

The  "Types  of  Plans"  screens  were  a  critical  component  of  the 
website.  This  section  helped  those  participants  who  were  privately 
insured  to  better  understand  the  costs  and  benefits  of  using  HMOs  versus 
PPOs.  In  evaluating  the  information  offered  on  HMOs  and  PPOs, 
participants  felt  that  the  information  provided  was  generally  accurate 
and  comprehensive  in  orienting  people  to  the  differences  between  these 
two  types  of  health  systems.  The  "Types  of  Plans"  screen  was  one  of  the 
two  screens  that  changed  depending  on  whether  the  user  was  insured  by 
Medicaid  or  a  private  company.  Those  who  were  privately  insured  were 
presented  with  information  differentiating  HMOs  and  PPOs.  Those  who 
received  Medicaid  were  simply  given  more  information  on  "What  Is  a 
Managed  Care  Plan?" 

During  the  first  round  of  testing,  participants  felt  that  the 
"Types  of  Plans"  description  of  in-network  and  out-of-network  costs  was 
clear  and  also  necessary  information  for  participants  to  understand  when 
trying  to  choose  a  health  plan.  "I  really  liked  the  way  that  this 
screen  did  a  side-by-side  comparison  on  HMOs  and  PPOs,  but  it  could  have 
been  clearer  if  the  HMOs  and  PPOs  were  also  compared  on  specific  points 
such  as  costs,  convenience,  in-network  vs.  out-of-network  costs."  Yet, 
participants  also  stated  that  the  structure  and  language  in  the  screen 
could  be  improved.  "You  can  see  here  that  some  of  the  information  is 
repeated.  In  both  the  HMO  and  PPO  sections  it  states  that  'You  may  also 
receive  better  coverage  for  some  services,  such  as  preventive  care.' 

How  can  that  be  the  case  for  both?"  Participants  had  additional 
questions  on  the  validity  of  the  screen  content.  For  instance,  the 
stated  advantages  of  HMOs  include  having  very  little  paperwork  and  co¬ 
payments  of  typically  $10.  As  one  participant  said  "There  needs  to  be 
more  cost  information  to  make  a  statement  like  this.  We  need  to  know 
about  co-payments  in  the  PPO  category  and  other  costs  that  are  factored 
in  to  having  one  of  these  two  types  of  plans." 
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screens 


No  significant  changes  were  made  to  the  "Types  of  Plans" 
during  the  second  round  of  testing,  for  either  the  privately  or  publicly 
insured  versions  of  the  site.  As  one  round  two  participant  said,  "I 
think  that  this  pretty  much  sums  up  some  of  the  main  differences  between 
HMOs  and  PPOs,  but  most  people  coming  to  look  at  a  plan  will  probably 
have  enough  experience  with  one  of  the  two  types  of  plans  to 
automatically  choose  one  or  the  other."  Participants  in  round  two  re¬ 
iterated  the  suggestion  that  statements  made  about  HMOs  should  be 
compared  to  information  about  PPOs.  Participants  stated  that  they  would 
have  liked  more  information  on  types  of  plans  before  making  a  final 
decision  on  a  plan  (Table  2) . 

SURVEY  DESCRIPTION  AND  TOPICS 

Participants  felt  that  the  "Survey  Description"  screen  was 
critical  to  the  site's  credibility.  Understanding  who  conducted  the 
survey,  who  the  survey  respondents  were  and  where  they  were  located  was 
to  all  of  the  interviewees .  Establishing  trust  in  the  survey 
was  necessary  for  participants  to  find  the  graphs  and  charts  valid  and 
the  information  personally  relevant.  As  one  participant  said,  "I  had 
better  trust  where  this  information  is  coming  from  or  else  I  won't 
believe  it."  Unfortunately,  many  of  the  first  round  participants  did 
not  believe  that  the  survey  was  done  in  an  objective  manner.  Four 
participants  in  round  one  stated  that  the  company  sponsoring  the  survey 
of  its  employees  had  the  opportunity  to  change  the  results  to  favor  a 
certain  health  plan.  After  looking  through  all  of  the  survey 
information,  participants  were  asked  to  recall  who  conducted  the  survey 
and  how  many  people  participated  in  it.  Although  this  information  was 
clearly  stated  on  the  "Survey  Topics"  screen,  five  of  the  round  one 
participants  could  not  recall  the  number  of  participants  and  four  could 
not  recall  who  sponsored  the  survey.  After  reviewing  the  screen, 
participants  were  asked  who  they  felt  paid  for  the  survey.  Seven 
interviewees  felt  it  was  paid  for  by  an  independent  research 
organization,  two  thought  the  health  plans  paid  for  it,  and  one  didn't 
know.  Additionally,  when  round  one  participants  were  asked  whether  the 
survey  results  were  accurate  representations  of  what  the  health  plans 
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were  "really  like,"  four  said  that  they  were  not,  four  didn't  know,  and 
only  three  thought  that  they  were .  After  they  understood  that  an 
independent  research  group  conducted  the  survey,  several  participants 
said  they  liked  the  fact  that  the  information  was  coming  from  a 
consumers'  rather  than  a  health  plan  or  employers'  point  of  view.  As  one 
woman  said,  "It  makes  a  big  difference  where  we  think  that  the 
information  is  coming  from.  If  it's  from  my  health  plan  I'm  less  likely 
to  trust  it  than  if  it  is  from  a  real  research  group." 

To  increase  the  trustworthiness  of  the  survey  results, 
participants  suggested  that  information  on  the  survey  sponsor  and  the 
demographic  make-up  of  the  survey  participants  should  be  more  clearly 
stated  on  the  screen,  either  by  placing  this  information  in  bold,  or  by 
moving  it  to  the  top  of  the  screen.  If  the  demographic  characteristics 
of  the  sample  were  made  clear,  participants  felt  that  they  would  be  more 
likely  to  get  an  accurate  picture  of  how  their  own  experience  with  their 
health  plan  was  likely  to  compare  with  that  of  the  survey  participants. 
An  additional  suggestion  was  to  improve  the  description  of  the  survey 
population.  Some  participants  were  unsure  whether  the  300  adults 
mentioned  on  the  "About  the  Survey"  screen  evaluated  each  of  the  health 
plans  or  only  one  plan,  and  how  many  survey  participants  rated  each 
topic  in  each  health  plan.  Two  participants  mentioned  that  they  would 
have  liked  the  health  plans  to  be  rated  by  physicians  as  well  as 
patients . 

In  order  to  emphasize  that  the  survey  was  done  in  an  objective 
manner,  the  information  on  who  conducted  the  survey  and  how  it  was  done 
was  moved  to  the  top  of  the  "Survey  Results"  screen  in  the  revised 
version  for  the  Decision  Helper.  The  text  explaining  the  survey  was 
made  more  visible  by  increasing  text  spacing  and  increasing  text  font. 
The  fact  that  the  survey  was  conducted  by  an  independent  research 
company  and  not  by  the  health  plans  was  added  to  the  "Welcome"  screen. 

We  did  not,  however,  change  the  actual  description  of  the  survey. 

The  changes  made  to  the  "About  the  Survey"  section  of  the  site 
improved  users'  understanding  of  who  the  survey  respondents  were.  Eight 
of  the  eleven  round  two  participants  were  able  to  describe  the  number  of 
survey  respondents  and  show  where  this  information  had  been  presented  on 
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the  site.  Unfortunately,  the  changes  did  not  seem  to  help  participants 
understand  that  the  survey  was  conducted  by  an  independent  research 
organization.  Three  of  the  eleven  round  two  participants  thought  that 
the  health  plans  sponsored  the  survey,  five  thought  that  an  independent 
research  organization  paid  for  the  survey,  two  thought  a  government 
agency  paid  for  it,  and  one  did  not  know. 

Each  of  the  participants  in  both  rounds  of  testing  thought  that 
the  survey  topics  were  important  issues  pertaining  to  health  plan 
choice.  The  "Survey  Topics"  screen  served  as  an  introduction  to  the 
seven  health  plan  issues  that  would  be  discussed  throughout  the  site. 

The  only  problem  that  first  round  participants  had  with  this  screen  was 
difficulty  in  seeing  the  on-screen  links  that  would  lead  to  either  the 
"Summary"  or  "Details"  screen.  As  a  result,  these  links,  like  all 
others  throughout  the  site,  were  changed  from  black  to  blue  so  that  they 
would  be  more  visible. 

OTHER  SURVEY  RESULTS 

The  "People  Who've  Used  a  Lot  of  Care"  and  "Children's  Care" 
screens  were  used  much  less  frequently  than  the  screen  showing  the 
survey  results  for  adults.  There  may  be  several  reasons  for  this. 
Participants  mentioned  that  categorizing  children's  care  and  frequent 
care  under  "Other  Survey  Results"  made  it  seem  as  if  these  topics  were 
less  important  or  that  these  sections  would  simply  review  the  results 
that  had  already  been  presented.  As  a  result,  visits  to  these  screens 
using  the  Contents  Box  were  limited.  In  addition,  if  participants  went 
to  the  "Other  Survey  Results"  screen  using  the  arrow  keys,  they  were 
likely  to  miss  the  on-screen  links  to  the  "Children's  Care"  and  "People 
Who've  Used  a  Lot  of  Care"  bar  graphs. 

Additional  concerns  were  raised  about  how  "People  Who  Have  Used  a 
Lot  of  Care"  was  defined.  All  eleven  of  the  first  round  participants 
felt  that  three  physician  visits  per  year  did  not  constitute  "a  lot  of 
care".  Participants  suggested  that  the  survey  title  be  changed  to  more 
accurately  reflect  the  amount  of  care  survey  respondents  reported. 
Although  none  of  the  round  one  participants  or  anyone  covered  on  their 
health  plan  needed  a  lot  of  care,  several  stated  that  if  it  had  been 
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more  clear  what  information  would  be  presented  on  this  screen,  they 
would  have  viewed  the  screen.  As  one  participant  observed,  "These 
people  really  know  what  the  plans  are  like  because  they  use  them  a  lot. 

I  would  probably  trust  these  graphs  more  than  the  others."  Navigation 
to  the  "Children's  Care"  screens  was  fraught  with  problems  similar  to 
navigation  to  the  "People  Who've  Used  a  Lot  of  Care"  screen.  Not 
surprisingly,  the  participants  who  felt  the  "Children' s  Care"  screen  was 
most  useful  were  those  who  had  children.  Three  participants  mentioned 
that  they  thought  that  screens  targeting  specialties  were  a  good  idea. 

As  one  woman  said,  "I  would  really  like  it  if  I  could  personalize  the 
site.  For  example,  if  I  could  choose  information  on  oncology,  AIDS 
care,  or  gynecological  services  and  the  ratings  would  show  how  the 
health  plans  supported  these  types  of  services." 

As  mentioned  previously,  changing  the  format  of  the  Contents  Box 
significantly  improved  navigation,  usability  and  usefulness  of  the 
"Children's  Care"  and  "People  Who've  Used  a  Lot  of  Care"  screens. 

Because  the  second  version  of  the  Contents  Box  listed  both  screens 
rather  than  listing  them  under  the  sub-heading  of  "Other  Survey 
Results,"  participants  had  an  easier  time  locating  these  screens. 
Additionally,  "People  Who've  Used  a  Lot  of  Care"  was  changed  to  "People 
With  3+  Visits"  in  the  revised  version.  In  terms  of  usefulness  of  these 
screens,  participants  in  the  second  round  expressed  similar  sentiments 
to  first  round  participants.  Individuals  with  children  found  the 
"Children's  Care"  screens  most  useful,  and  many  participants  felt  that 
the  "People  with  3+  Visits"  was  especially  useful  because  these  people 
would  have  ample  experience  in  dealing  with  physicians  and  health  plans . 

SUMMARY  AND  DETAIL  CHARTS 

Overall,  users  found  the  "Summary"  charts  to  be  more  usable  but 
less  useful  than  the  "Detail"  charts.  Although  people  presumed  that  the 
stars  were  used  to  rate  the  plan  characteristics  rather  than  to  compare 
them  to  the  survey  average,  they  felt  that  the  "Summary"  charts  were 
easier  to  read,  understand  and  use.  Participants  felt  that  they  could 
understand  the  "Summary"  charts  without  extra  instruction. 
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Consequently,  although  16  participants  missed  the  '^What  Do  the  Stars 
Tell  You?"  screen,  13  felt  that  this  screen  was  not  necessary, 

Web  site  users  found  the  bar  graphs  on  the  "Details"  screen  more 
useful  than  the  "Summary"  star  charts  and  more  integral  to  understanding 
the  survey  results.  Five  of  the  22  participants  did  not  understand 
whether  the  graphs  were  charting  all  of  the  characteristics  of  all  of 
the  plans  (a  comprehensive  percentage  rating  of  each  of  the  plans  on 
overall  qualities)  or  each  of  the  bullet  points  for  each  of  the  plan 
characteristics  (e.g.,  under  the  doctor  communication  rating,  the 
bullets  would  be  doctor's  ability  to  communicate  clearly,  spend  enough 
time  with  the  patients,  show  respect  for  patients,  etc.).  Yet  once  they 
understood  what  the  bar  graphs  were  showing,  they  found  the  information 
more  valuable,  stating  that  the  "percentages  quantified  people's 
opinions"  and  that  the  "bar  graphs  were  more  useful  because  they  gave 
specific  numbers  rather  than  averages." 

One  aspect  of  the  website  that  could  have  led  to  the  bar  graphs 
being  seen  as  more  difficult  to  use  and  understand  was  that  five  of  the 
round  one  and  two  round  two  participants  missed  the  on’-screen  link  to 
the  "What  Do  the  Bar  Graphs  Tell  You?"  due  to  difficulty  locating  the 
links  to  this  screen.  When  interviewees  were  exposed  to  this  screen,  all 
but  one  said  that  they  would  have  found  this  information  useful,  as  it 
explained  how  to  read  the  bar  graphs  as  well  as  how  the  star  chart  and 
bar  graphs  were  connected.  Two  participants  were  unsure  whether  the 
star  charts  and  bar  graphs  were  both  using  data  from  the  survey;  this 
screen  clarified  this  point. 

When  comparing  the  usefulness  of  the  "Summary"  and  "Detail" 
charts,  12  users  preferred  the  "Detail"  charts,  and  10  preferred  the 
summary  charts  {See  Figure  5) .  In  order  to  combine  the  best  aspects  of 
these  charts,  two  users  suggested  combining  the  two  by  having  a  single 
chart  with  all  the  bar  graph  percentages. 

REVIEW  THE  RESULTS  SCREEN 

The  "Review  the  Results"  screen  was  key  to  helping  users  summarize 
the  most  personally  valuable  information  presented  on  the  site  and  to 
input  this  information  for  use  on  the  "Your  Worksheet"  screen.  In  Round 
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I,  eight  participants  were  able  to  tell  the  difference  between  the 
"Review  the  Results"  and  the  "Summary"  screens  at  first  glance.  There 
were  three  separate  components  to  this  screen:  plan  evaluations,  a 
summary  table,  and  a  total  stars  composite  score.  Because  participants 
had  been  presented  with  the  "Summary"  charts,  which  used  stars  to 
evaluate  plan  quality,  they  were  confused  about  the  difference  between 
the  "Review  the  Results"  and  "Summary"  screens.  Additionally  some 
participants  did  not  realize  that  if  they  clicked  the  "Click  Here" 
button,  plan  quality  results  would  be  summarized.  Additionally, 
participants  did  not  feel  that  the  purpose  of  inputting  information  was 
explained.  As  a  result,  these  participants  did  not  realize  that  the 
"Did  the  Plan  Do  Well...."  questions  would  link  to  the  "Worksheet"  screen. 
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Figure  5:  Participant  Preference,  of  Star  versus  Bar  Charts 


When  discussing  usability  issues,  some  participants  stated  that 
Internet  users  seldom  needed  to  read  instructions  to  figure  out  how  to 
navigate  through  an  individual  screen.  As  a  result,  people  were  less 
likely  to  take  the  time  to  read  screen  instructions.  When  re-examining 
the  "Review  the  Results"  screen,  some  participants  hypothesized  that, 
because  of  the  shaded  background,  they  were  less  likely  to  notice  the 
instructional  text.  Seven  of  the  participants  did  not  realize  that 
their  answer  on  the  "Did  the  Plan  Do  Well..."  question  would  link  to  the 
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"Your  Worksheet"  screen.  These  people  found  the  Review  the  Results 
screen  valuable  and  appreciated  that  they  could  personalize  some  of  the 
screens.  Yet  even  these  participants  mentioned  that  the  link  between 
responses  on  the  "Review  the  Results"  screen  and  the  "Worksheet"  screen 
should  be  made  clearer  because  it  could  easily  be  missed. 

Some  participants  did  not  see  why  it  was  necessary  to  use  the 
"Click  Here"  button  to  find  the  total  number  of  stars.  They  felt  that 
the  star  totals  should  be  automatically  presented  on  this  screen. 
However,  three  participants  stated  that  it  was  detrimental  to  the  health 
plan  decision-making  process  to  have  the  total  nioinber  of  stars  for  each 
plan  available.  If  totals  are  presented,  individuals  will  be  less 
likely  to  explore  each  of  the  topics  that  made  up  the  total  score. 

Users  will  be  tempted  to  choose  a  plan  based  on  total  score  rather  than 
examining  and  weighting  the  value  of  each  health  plan  characteristic. 

As  one  participant  stated,  "By  just  giving  people  the  total  score,  they 
won't  look  at  all  of  the  components  that  make  up  that  score." 

To  remedy  problems  with  the  instructions  on  this  screen,  the  web 
designers  inserted  a  pop-up  box  that  appeared  with  the  "Review  the 
Results"  screen.  This  box  included  information  on  the  interactive  nature 
of  the  screen  (results  would  link  with  the  "Your  Worksheet"  screen) . 
Participants  had  to  click  on  this  small  screen  and  answer  the  questions 
on  the  "Review  the  Results"  screen  to  move  onto  the  next  screen. 

Although  participants  with  more  computer  experience  and  those  who  were 
able  to  see  the  links  on  the  "Review  the  Results"  screen  found  this  pop¬ 
up  box  somewhat  annoying  ("It  reminds  me  of  those  advertising  boxes  that 
always  show  up  on  websites"),  these  pop-up  screens  aided  some  users  in 
better  understanding  and  utilizing  the  "Review  the  Results"  screen. 
Participants  with  less  experience  especially  appreciated  the  pop-up  box, 
saying  that  the  instructions  "made  it  clear  what  I  had  to  do  on  the 
screen . " 

Only  two  of  the  first-round  participants  found  this  screen 
(without  the  pop-up  instructional  boxes)  very  useful,  versus  six  of  the 
second-round  participants .  Six  of  the  first-round  and  three  of  the 
second-round  participants  found  the  screen  somewhat  useful.  Two  of  the 
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first-round  and  two  of  the  second-round  participants  found  it  slightly 
useful  (see  Figure  6) . 


Not  at  all  Slightly  Somewhat  Very 

Rating  of  Page 


■  Round  I  11  Round  II 


Figure  6:  Usefulness  of  "Review  the  Results"  Screen 


YOUR  WORKSHEET 

Navigation  to  the  "Your  Worksheet"  screen  was  relatively  easy  for 
participants  in  both  rounds  of  testing.  Because  participants  could 
utilize  the  Contents  Box  or  the  arrow  keys  to  reach  this  point  in  the 
site,  none  skipped  the  screen  during  their  free  exploration. 

Participants  thought  it  was  very  beneficial  to  have  a  worksheet  screen 
on  which  their  reactions  to  the  site  and  their  thoughts  on  health  plan 
quality  could  be  placed.  Yet  they  would  have  liked  the  ability  to 
complete  the  boxes  on  the  "Worksheet"  screen  with  their  own  information 
to  make  the  screen  more  interactive  and  more  personal.  Ten  of  the  22 
round  one  and  two  participants  stated  that  they  would  like  to  be  able  to 
enter  notes  on  the  plan  characteristics  alongside  the  "Your  Worksheet" 
plan  ratings.  "It  would  be  great  if  I  could  make  notes  to  myself  about 
these  different  plans  and  it  seems  like  that  is  the  purpose  of  the 
"Worksheet"  screen.  But  we  can't  write  on  the  screen!!" 

In  the  first  round  of  testing,  some  participants  were  confused  by 
the  underlined  question  marks  on  the  "Your  Worksheet"  screen.  Because  a 
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majority  of  the  participants  did  not  respond  to  the  interactive 
questions  on  the  "Review  the  Results"  screen,  either  on  the  general  or 
Other  Survey  Results"  screens,  the  question  mark  signified  that  there 
was  an  unanswered  topic.  A  pop-up  box  was  added  to  the  "Worksheet" 
screen  to  help  users  understand  that  the  question  marks  are  actually  on¬ 
screen  links.  The  text  on  this  screen  explained  the  use  of  links  to 
personalize  the  "Your  Worksheet"  screen  and  the  fact  that  the  links 
would  bring  participants  back  to  the  information  that  they  did  not 
complete  on  the  previously  viewed  screens.  About  half  the  interviewees 
stated  that  they  would  print  out  the  "Worksheet"  screen  if  they  were 
using  the  Decision  Helper  to  choose  a  health  plan.  Others  felt  that  the 
screen  s  usefulness  was  limited  unless  it  could  be  made  more  personal. 

DOCTORS,  COSTS  AND  BENEFITS 

Thirteen  of  the  22  participants  did  not  see  the  "Your  Worksheet" 
screen's  on-screen  links  to  information  on  doctors,  costs  and  benefits. 
Changing  the  color  of  the  links  to  stand  out  from  the  text  helped  users 
in  round  two  to  see  and  utilize  them.  During  the  screen— by— screen 
website  assessment,  ten  of  the  interviewees  were  initially  disappointed 
with  the  Doctors,"  "Costs"  and  "Benefits"  screens.  They  had  expected 
specific  information  about  the  doctors  that  the  plan  offered,  specific 
costs  such  as  for  tests  and  prescriptions,  and  benefits  covered  by  each 
plan.  However,  after  reading  the  screens,  every  participant  agreed  that 
the  information  was  valuable,  easy  to  read,  and  provided  crucial 
information  to  consider  when  choosing  a  health  plan.  "This  was  the  best 
part  of  the  whole  site,"  said  one  participant.  "These  are  the  questions 
that  people  should  be  thinking  about  when  choosing  a  plan.  This  is 
concrete  information  that  I  can  use  no  matter  what  type  of  health  care  I 
choose."  Participants  appreciated  the  objective  nature  of  the 
information  because  they  were  able  to  apply  it  to  their  current  plans 
and  their  future  health  plan  decisions. 

GETTING  MORE  INFORMATION 

From  the  "Your  Worksheet"  screen,  users  had  little  trouble 
navigating  to  the  "Getting  More  Information"  screens.  The  participants 
felt  that  the  suggestions  for  users  listed  on  these  screens  were,  on  the 
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whole^  very  reasonable.  Going  to  visit  one's  doctor's  office  and 
talking  to  one's  doctors  were  felt  to  be  necessary  when  making  an 
educated  health  plan  choice.  Ten  of  the  first-  and  second-round 
participants  stated  that  they  would  have  preferred  being  linked  to  the 
health  plan  websites  rather  than  just  being  given  the  plan  phone 
numbers.  They  would  have  found  it  especially  useful  if  they  could 
enroll  in  their  selected  health  plan  on-line.  In  this  way,  they  would  be 
able  to  avoid  "the  most  awful  part  of  the  health  plan  decision 
process .. -waiting  on  the  phone  for  the  health  plan  operators." 

Site  users  felt  the  questions  listed  on  the  "Making  Your  Choice" 
screen  were  important.  Yet  because  the  website  encouraged  users  to 
consider  the  logistics  of  different  plans  participants  once  again 
reiterated  the  need  for  more  factual  plan  information.  For  example, 
"Which  plans  have  doctors  you  like?"  and  "Which  plans  can  you  afford?" 
were  questions  that  need  to  be  answered  when  choosing  a  plan,  but  these 
questions  were  not  answered  by  the  information  provided  on  the  site. 

Users  particularly  appreciated  the  "Getting  More  Information" 
screens  because  the  questions  could  be  applied  to  the  current  health 
care  decision-making  they  were  doing.  Understanding  what  to  look  for 
when  choosing  a  plan  was  information  that  they  could  take  from  the 
interview  and  use  in  their  everyday  life.  As  one  participant  said, 
"Those  are  good  questions  to  ask  and  ones  that  I  hadn't  really  thought 
of  before." 

The  "Summary"  and  "Detail"  screens  are  designed  to  help  users  make 
a  more  informed  decision  than  they  might  otherwise  have  been  able  to 
make  about  the  best  health  plan  for  themselves.  Participants  mentioned 
information  on  hospital  locations,  co-payments,  difficulty  in  getting 
referrals,  and  quality  of  specialists  as  information  needed  before 
making  a  final  plan  decision.  Yet  even  without  this  information,  16  of 
the  round  1  and  2  participants  felt  that  they  could  eliminate  certain 
plans  using  only  the  information  given  on  the  Decision  Helper. 
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CONCLUDING  REMARKS 


To  assess  the  usability  and  usefulness  of  the  Decision  Helper 
version  2.0,  we  solicited  the  reactions  and  opinions  of  22  individuals. 
Eleven  individuals  who  participated  in  the  first  round  of  usability 
testing  helped  us  to  identify  a  few  key  elements  that  needed 
improvement.  This  included  making  site  navigation  more  obvious  and  more 
fluid,  changing  aspects  of  site  design  including  background,  font  size 
and  text  color,  explaining  the  interactive  nature  of  the  site,  improving 
some  of  the  language  used,  and  adding  more  objective  information. 

The  overall  site  design  and  navigation  tested  well,  and 
improvements  were  made  on  just  a  few  of  the  areas  noted:  the  Contents 
Box  and  links  were  given  more  explanation  and  made  more  obvious,  the 
background  colors  and  font  were  changed  to  be  more  aesthetically 
pleasing,  more  visible,  and  easier  to  read,  and  pop-up  boxes  were  added 
on  two  of  the  screens  to  explain  the  interactive  nature  of  those 
portions  of  the  site.  We  were  not  able  to  address  all  usability 
problems  identified  in  the  first-round  of  testing,  either  because  the 
CAHPS®  template  did  not  allow  for  information  to  be  added  or  because  the 
content  needed  to  remain  consistent  with  other  CAHPS®  reporting 
materials.  Though  we  did  expand  on  certain  areas  of  the  tool's  content, 
adding  more  explanations  and  highlighting  important  text,  we  did  not 
change  the  core  content  of  any  screens . 

Participants  found  the  limited  amount  of  information  on  the  site 
frustrating.  Although  they  understood  the  site's  objective  and  could 
navigate  through  it  fairly  well,  they  felt  they  would  have  had  a 
difficult  time  using  it  to  choose  a  health  plan.  Without  information  on 
the  doctors,  costs  and  benefits  of  the  specific  plans,  none  of  the 
participants  felt  that  they  could  draw  conclusions  on  the  best  plan  for 
them.  In  the  absence  of  this  information,  participants  felt  that  the 
"Your  Worksheet"  screen  had  limited  value,  merely  serving  to  emphasize 
the  frustration  of  not  being  able  to  use  the  site  to  pull  together  all 
the  relevant  information.  Although  participants  were  able  to  rate  plans 
based  on  the  topics  they  found  most  relevant,  they  were  not  able  to 
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integrate  CAHPS®  survey  information  with  other  plan-related  information 
in  making  a  health  plan  decision.  Because  employers  do  not  all  have  the 
same  type  or  amount  of  information  available,  a  template  for  how  such 
information  should  be  included  in  the  Decision  Helper  would  be  of 
limited  value.  Accordingly,  we  tested  a  version  of  the  Decision  Helper 
that  included  only  CAHPS®  information.  However,  the  web  site  has  been 
designed  in  such  a  way  that  additional  information  on  benefits,  costs, 
doctors,  and  HEDIS  or  other  quality  measures  can  be  added  with 
relatively  little  modification  to  the  parts  designed  to  present  CAHPS® 
information  and  provide  decision  support  for  choosing  a  health  plan. 
Employers  can  and  should  make  the  site  more  useful  by  including  as  much 
information  as  possible  bearing  on  health  plan  choice. 

An  additional  finding  was  that  having  basic  Web  experience  appears 
to  be  critical  for  utilizing  the  full  potential  of  the  Decision  Helper. 
Our  revisions  did  not  include  any  attempt  to  make  the  site  more  usable 
for  those  without  computer  experience,  and  we  are  not  at  all  confident 
that  such  an  attempt  would  be  successful.  First  and  second  round 
participants  who  had  not  used  the  Web  did  not  feel  comfortable  using  the 
computer  as  a  medium  to  make  health  care  decisions.  The  barriers  to 
successful  use  of  the  Decision  Helper  by  those  without  computer 
experience  are  sufficiently  formidable  that  it  is  probably  wise,  as  a 
practical  matter,  to  provide  only  print-based  informational  materials 
for  consumers  in  this  category. 
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APPENDIX  I.  ROUND  1  PROTOCOL 


Task  1:  Interviewer  asks  subject  to  freely  explore  Decision  Helper,  as 
she  would  if  comparing  health  plans.  Interviewer  is  an  unobtrusive 
observer  and  ready  to  answer  questions.  Interviewer  takes  notes, 
(approximately  30  minutes) 

Task  2:  Interviewer  asks  subject  to  comment  on  any  and  all  aspects  of 
the  site.  If  necessary,  prompt  with  "what  did  you  like  or  dislike  about 
individual  screens  or  the  site  as  a  whole?"  (approximately  30 
minutes ) 

Task  3:  Interviewer  conducts  structured  interview.  Interviewer  first 
asks  subject  about  individual  screens,  then  overall  site. 

(approximately  35  minutes) 

Task  4:  Interviewer  asks  subject  to  complete  written  questionnaire, 
(approximately  20  minutes) 

Subject  #  _ 

Start  time  _ 

End  time  _ 

Computer  station  _ 


Hi,  my  name  is  Mina  Kimmerling  and  I  work  for  the  RAND 
Corporation,  which  is  a  research  organization.  Today,  I'm  going  to  show 
you  a  website  which  is  used  to  help  people  compare  the  quality  of  health 
plans.  I'd  like  to  get  your  reactions  to  this  website,  which  is  called 
"Decision  Helper." 

I'm  going  to  ask  you  to  look  over  the  website  and  then  I'm  going 
to  ask  you  a  series  of  questions  about  your  impressions .  I  want  you  to 
know  that  there  are  no  right  or  wrong  answers.  This  is  not  a  test  of 
your  abilities,  but  is  a  test  of  the  computerized  decision  helper.  From 
time  to  time  a  may  interrupt  you  to  ask  you  a  few  questions.  We  want  to 
know  what  you  like  and  dislike  about  the  Decision  Helper,  so  that  the 
people  who  designed  it  can  improve  it.  We're  going  to  spend  about  90 
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minutes  looking  through  the  site,  depending  on  how  you  use  the  Decision 
Helper.  If  you  need  to  take  a  break  at  any  time,  let  me  know. 

We  want  to  learn  what  you  think  can  be  improved,  especially  the  parts 
that  are  the  most  confusing  or  the  most  easy  to  understand.  So  please  be 
very  honest  in  your  answers. 

I  want  to  reassure  you  that  we  will  keep  the  information  you  give 
us  confidential.  Your  name  will  not  be  written  in  my  notes.  Your  name 
will  not  appear  in  the  report  we  write  summarizing  these  test  sessions. 
You  have  the  right  not  to  answer  any  question  and  you  are  free  to  stop 
participating  at  any  time.  Of  course,  I  would  like  you  to  participate 
to  the  fullest  that  you  can.  I'd  like  you  to  use  it  as  you  would  in  your 
own  home  or  office  to  choose  a  health  plan  for  you  and  your  family. 
Before  we  begin,  let  me  just  go  over  a  couple  things  on  the  computer  (If 
needed  show  R  how  to  navigate  screens,  use  mouse,  etc.) 

Let  me  also  ask  you  a  couple  questions. 

Do  you  have  children  that  are  covered  by  your  health  plan? 


Do  you  have  anyone  who  needs  a  lot  of  care  covered  by  your  health  plan? 


Today's  interview  will  take  approximately  2  hours.  Our  session 
today  will  have  three  parts.  First,  I'm  going  to  have  you  look  through 
the  "Decision  Helper"  website  as  if  you  were  using  it  to  choose  a  health 
plan  for  yourself.  After  you  have  looked  at  the  site,  I'm  going  to  go 
through  the  site  with  you  and  get  your  impressions  on  each  screen.  I 
also  want  you  to  share  with  me  any  ideas,  questions  or  suggestions  that 
you  might  have  about  any  part  of  the  decision  helper  website. 

It  is  important  that  we  hear  what  you  are  thinking,  what  you  see,  and 
how  useful  you  find  each  screen.  I'd  like  to  hear  what  you  think  about 
each  section  of  the  site  after  you  have  had  a  chance  to  use  it.  So  as 
we  begin  why  don't  you  go  give  me  your  spontaneous  reactions  and  candid 
thoughts  to  each  screen.  As  you  look  through  the  site,  you  can  read 
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each  section  as  if  you  were  going  through  a  restaurant  menu,  talking  out 
loud  about  the  specific  and  general  aspects  of  the  site  that  you  notice. 

FREE  EXPLORATION 

Take  as  long  as  you  want  to  go  through  the  site.  You  don't  have 
to  go  through  the  site  in  any  particular  order.  Take  as  long  as  you 
want  to  go  through  the  site.  You  don't  have  to  go  through  in  any 
particular  order.  Just  use  your  judgment  and  let  me  know  why  you  skip 
around  and  what  type  of  order  you  are  going  in. 

Are  you  ready  to  begin?  Okay,  you' 11  start  on  the  Splash  Screen,  which 
is  the  one  already  loaded. 

Mark  Begin  Time _ 


Interviewer  should  unobtrusively  observe  and  take  notes  on  subject's 
reactions  to  website. 

Amount  of  time  spent  on: 

■  Introductory  screens 

■  Star  explanation 

■  Star  screens 

■  Bar  explanation 

■  Bar  Screens 

■  Child  Care  information 

■  "People  who've  used  a  lot  of  care"  information 


Is  there  a  screen  participant  spends  a  lot  of  time  on?  (ASK  PERTINENT 
FOLLOW-UP  QUESTIONS) 

Seems  particularly  confused  by? 
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Continuously  flips  back  and  forth  to? 

Glances  at,  but  skips  over? 

Seems  to  miss? 

Is  there  a  part  of  the  screen  that  eyes  are  immediately  drawn  to? 

Is  there  an  obvious  sequence  to  participant  searching  patterns? 
Note  order  of  screens 

Navigation  techniques -what  do  they  use  to  get  around  the  site? 
Does  participant  make  use  of  the  final  screen? 

Flip  to  see  total  number  of  stars? 

Mark  which  items  they  find  most  useful? 

Use  the  final  tally  sheet  for  gathering  more  information? 


Mark  End  Time^ _ 

UNSTRUCTURED  INTERVIEW 

I'd  like  to  begin  by  having  you  give  your  overall  impressions  of  the 
site.  What  you  liked,  what  you  didn't  like,  etc. 


After  a  participant  gives  general  overview,  proceed  with  these 
questions : 

1.  Overall,  was  it  easy  or  hard  to  use? 

2.  Overall,  useful  was  the  information? 

3.  What  did  you  like  most  about  Decision  Helper? 

4.  What  did  you  like  least? 
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5.  Do  you  feel  like  you  learned  a  lot  by  going  through  it?  Was  the 
information  useful? 


6.  Do  you  think  that  this  is  the  type  of  site  that  you  would  have  to  go 
through  a  few  times  to  get  the  most  out  of  it?  Is  it  difficult  to 
use? 


7.  After  looking  through  the  entire  site  how  important  is  this  survey 
information  to  you  in  deciding  upon  a  health  plan? 


8.  Who  do  you  think  would  use  this  site?  Is  this  site  easy  to  use 
without  explanation  or  do  you  think  that  doesn't  have  computer 
experience  would  have  a  hard  time  using  it? 


9.  What  would  make  it  easier  to  use? 


10. What  could  you  see  people  using  the  site  for  in  terms  of  comparing 
plans,  learning  about  different  types  of  plans,  choosing  a  plan? 

STRUCTURED  INTERVIEW 

Now,  we're  going  to  go  through  each  of  the  screens  of  the  website.  I 
want  to  walk  through  the  site  going  from  screen  to  screen  in  the  order 
that  you  went  through  it.  On  each  screen,  I'd  like  you  to  mention 
anything  you  found  especially  confusing  or  especially  useful.  I'd  also 
like  you  to  tell  me  what  you  think  the  purpose  of  each  screen  is,  or 
what  information  is  being  conveyed.  (At  the  end  of  each  section  ask  if 
there  is  anything  else  they  would  like  to  add  or  any  questions  they  may 
have . ) 
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(Note  what  navigation  tools  participant  uses  to  move  from  one  screen  to 
the  next . ) 

(See  if  people  click  to  expand  on  the  major  topics  to  see  subtopics  or 
just  move  on  to  next  major  heading.  Check  if  this  is  a  reason  that  they 
are  skipping  screens.) 

Questions 

For  all  screens  that  participants  missed  go  through  how  they  could  have 
gotten  there  and  whether  this  was  a  screen  that  they  would  have  found 
useful . 

Splash  Screen 

1.  What  is  your  overall  reaction  to  this  screen?  What  things  did  you 
notice  about  it? 

2.  The  screen  says,  "A  Guide  to  Health  Plan  Quality  from  the  Consumer 
Point  of  View."  Did  you  notice  that?  What  did  it  mean  to  you? 

3.  What  do  you  think  about  when  you  look  at  the  picture  on  this  screen? 
Did  you  wonder  whether  the  people  are  connected  to  the  Decision 
Helper  in  some  way? 

4.  Say  you  were  just  browsing  on  the  web  and  came  to  this  screen.  After 
looking  at  it,  do  you  think  that  you  would  click  "start"  to  see  more, 
or  would  you  be  more  likely  to  continue  browsing?  Why? 


Welcome  section 

1.  What  did  you  look  at  on  this  screen?  What  information  did  you  get 
from  it? 

2.  What  does  the  information  in  this  box  (point  to  table  of  contents) 
tell  you? 

3.  Did  you  use  this  list  to  help  you  navigate  around  the  site  (as  a 
menu) ?  How  confusing  was  it  to  use? 
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4.  How  clear  did  you  find  the  information  in  the  table  of  contents?  Did 
you  understand  that  you  could  switch  web  screens  by  clicking  on  a 
topic? 

5.  Did  the  content  of  the  screens  reflect  their  description  in  the  table 
of  contents.  For  example  was  the  "Review  the  Results"  screen  what  you 
expected? 

6 .  Were  the  topics  covered  the  ones  that  you  find  useful  in  choosing  a 
health  plan? 

7.  After  looking  through  the  entire  "decision  helper"  website  do  you 
think  that  the  first  three  screens  are  a  good  introduction  to  the 
entire  site  or  is  there  something  else  that  you  would  like  to  see? 

8.  After  looking  through  the  entire  site,  do  you  think  that  the  survey 
description  accurately  explains  the  survey  of  the  different  health 
care  organizations? 

9.  Who  conducted  the  survey?  What  does  that  mean  to  you?  Is  it 
important  in  some  way? 


Navigation  tools 

1.  What  tools  did  you  use  to  navigate  around  the  site?  Was  it  easy  to 
figure  out  how  to  maneuver  around  the  site? 

2.  What  did  you  use  these  arrows  for?  Is  it  clear  that  they  are  used  to 
get  around  the  site? 

3.  Is  it  clear  that  if  you  click  on  the  next  keys  you  can  move  around 
the  site. 
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Types  of  plans 

1.  Have  you  heard  the  terms  listed  on  this  screen,  like  HMO  and  PPO 
before? 

2.  How  does  the  information  on  this  screen  compare  to  what  you  already 
knew  about  these  topics? 

3.  Do  you  think  that  the  information  on  this  screen  is  accurate? 

4.  Do  you  think  that  this  screen  provides  you  with  enough  information  to 
know  the  differences  between  health  plans? 

5.  What  type  of  information  would  help  clarify  the  differences  listed  on 
this  screen? 

6.  Was  it  important  to  you  to  understand  these  differences  when  choosing 
a  health  plan? 

7 .  Are  the  advantages  and  disadvantages  of  each  type  of  plan  useful  to 
your  decision  making  process? 

8.  Is  there  anything  on  this  screen  that  screen  is  confusing,  such  as 
the  screen  layout  or  the  language  used? 

9.  Was  it  easy  to  find  this  screen  while  navigating  around  the  site? 


About  the  Survey  Screen 

What  parts  of  this  screen  did  you  read?  What  do  you  think  the 
information  on  this  screen  in  supposed  to  do? 

1.  What  did  you  want  to  know  about  the  survey?  Does  this  screen  tell 
you  that? 
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2.  The  section  at  the  bottom  is  headed  "Can  you  trust  these  survey 
results?"  Does  the  text  underneath  tell  you  what  you  want  to  know 
about  that?  Why  or  why  not? 

3.  How  important  is  it  to  you  to  know  how  the  survey  was  done? 


Survey  results  screen 

1 .  How  helpful  were  the  survey  results  in  choosing  a  plan? 

2.  Is  this  screen  clear  in  explaining  the  survey  and  the  participants  in 
the  survey? 

3.  Do  you  feel  that  your  experiences  with  a  particular  health  plan  will 
be  similar  to  the  experiences  of  the  survey  participants? 

4.  Did  you  use  other  people's  experiences  in  helping  you  make  a  choice? 

5.  Do  you  see  a  difference  between  this  screen  and  the  reviewing  the 
results  screen? 

6.  Was  it  easy  to  find  this  screen  while  navigating  around  the  site? 

Survey  Topics  Screen 

1.  What  did  you  think  this  screen  was  for? 

2.  Did  you  read  the  list  of  survey  topics  that  people  were  asked  about? 
Which  of  these  topics  are  ones  that  you  would  expect  to  be  covered  in 
this  kind  of  report  about  health  plans?  Which  ones  would  you  expect 
not  to  be  covered? 

3.  Are  these  seven  topics  covered  on  this  screen,  the  most  important 
topics  to  you  in  choosing  a  health  plan? 
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4.  If  No,  what  topics  would  you  like  to  see  added?  Any  you  would  like 
to  see  dropped? 

5.  Do  these  topics  have  different  levels  of  importance  to  you?  Did  you 
go  through  the  different  topics  by  looking  at  which  was  the  most 
important  to  you? 

6.  Are  the  categories  mentioned  too  broad,  too  specific,  unclear  (etc.)? 

7.  At  the  bottom  of  the  screen  it  says  that  you  can  see  the  results  in 
two  formats?  What  did  this  mean  to  you?  Did  you  feel  like  you  knew 
what  you  were  choosing  when  you  clicked  on  "Summary"  or  "Details"? 

8.  Why  did  you  choose  to  go  to  summary  (or  details)  screen  first? 

(Note  whether  participant  goes  to  summary  or  details  screen  first. 

After  going  through  the  first  of  the  two,  do  they  return  to  this  screen 

to  see  the  second?  If  they  don't,  do  they  ask  for  more  information? 

Note  what  navigation  tool  participants  use  to  move  to  survey  summary  or 

details.  This  section  may  be  where  many  people  skip  sections.) 

9 .  Was  it  easy  to  find  this  screen  while  navigating  around  the  site? 

(If  people  have  skipped  this  screen,  show  them  how  they  skipped  and 
ask  whether  this  was  intentional . ) 


Summary  screen  (Stars) 

Note:  At  what  point,  if  any,  participants  click  on  the  "what  do  the 
stars  mean  screen."  How  much  time  is  spent  on  this  screen  relative  to 
details  screen,  survey  topics  screen  and  types  of  plans  screen?  How  much 
time  do  participants  spend  looking  at  star  meanings  on  top  right  hand  of 
screen? 

1 .  What  is  this  screen  supposed  to  do?  How  would  you  use  this 
information? 
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2.  Do  you  see  where  it  says  "What  do  the  starts  tell  you?"  Can  you  tell 
me  in  your  own  words  what  you  think  the  stars  tell  you? 

3.  When  you  look  at  the  results  listed  on  this  screen,  do  you  compare 
looking  at  both  PPOs  and  HMDs  or  did  you  look  at  the  plans 
separately? 

4.  What  did  the  number  of  stars  tell  you  about  each  health  plan  in 
relation  to  all  other  plans? 

5.  Do  you  think  that  you  could  choose  a  health  care  plan  by  just  looking 
at  the  information  on  this  screen?  If  no,  what  more  information 
would  you  need? 

6.  Do  you  think  that  the  information  on  this  screen  needs  to  be  more 
specific  to  individual  groups  (such  as  children,  the  elderly,  etc.)? 

7.  How  confusing  did  you  find  this  screen?  Which  parts  were  the  most 
confusing? 

8.  Was  it  easy  to  find  this  screen  while  navigating  around  the  site? 


Details  screen  (bar  graphs) 

Note:  At  what  point,  if  any,  do  participants  click  on  the  "what  do  the 

bars  mean  screen"?  How  much  time  do  participants  spend  looking  at  what 
do  bar  graphs  tell  you  screen?  (Are  they  noticing  this  link?)  How  much 
time  is  spent  on  bar  graph  descriptions  and  bar  graph  results  screen 
relative  to  summary  screen,  survey  topics  screen  and  types  of  plans 
screen? 

1.  What  do  you  think  this  screen  was  for?  How  would  you  use  this 
information? 
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2.  Can  you  explain  to  me  what  each  of  the  different  colors  of  the  bar 
are  supposed  to  tell  you? 

3.  Do  you  think  that  you  could  choose  a  health  care  plan  by  just  looking 
at  the  information  on  this  screen?  If  no,  what  more  information 
would  you  need? 

4.  Could  you  use  this  information  to  find  plans  that  you  would  NOT 
choose? 


5.  Do  you  feel  a  screen  summarizing  this  information  is  necessary? 

Could  you  make  do  without  it? 

6.  Do  you  think  that  this  screen  is  more  or  less  useful  than  the  summary 
screen  with  the  stars  on  it? 

7 .  Do  you  think  that  the  information  on  this  screen  needs  to  be  more 
specific  to  individual  groups  (such  as  children,  the  elderly,  etc.)? 
(If  they  say  yes,  note  whether  they  look  on  child  care  and  frequent 
user  information  screens) 

8 .  How  easy  or  hard  is  it  for  you  to  understand  the  information  on  this 
screen? 

9.  Is  there  anything  that  is  especially  confusing  or  some  area  that  you 
think  could  be  better  explained? 

10.  Was  it  easy  to  find  this  screen  while  navigating  around  the  site? 


Getting  the  Care  that  is  Needed 

1.  Are  the  topics  discussed  on  this  screen  clear  to  you? 

2.  Are  they  relevant  to  the  issues  you  think  about  when  choosing  a 
health  care  plan? 
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3.  How  useful  is  this  information  to  your  decision  on  which  health  care 
plan  to  choose? 

4.  How  do  you  think  each  of  the  plans  relate  to  each  other  in  terms  of 
quality  of  care  in  this  area  (can  you  tell  which  is  the  best,  which 
is  the  worst,  etc?) 

5 .  Do  you  think  that  the  information  given  by  the  survey  data  on  ability 
to  get  care  that  is  needed,  accurately  reflects  the  reality  of 
getting  care  that  is  needed? 

6.  How  does  the  information  on  this  screen  alter  or  effect  choosing  a 
health  care  plan? 

7.  Was  it  easy  to  find  this  screen  while  navigating  around  the  site? 


Review  the  Results 

1.  Do  you  feel  that  having  the  stars  and  bars  (summary  and  details 
sections  )  are  both  necessary  to  you  choosing  an  appropriate  health 
care  plan, 

2.  Which  do  you  prefer?  Why? 

3.  Does  that  mean  that  you  wouldn't  see  a  difference  if  one  section  was 
removed  from  the  website. 

4.  (See  if  choice  of  health  plans  is  different  using  only  stars  or  only 
bars?) 

5.  How  effective  is  this  screen  in  helping  you  make  a  health  care 
decision  from  this  screen? 

6.  Do  you  see  a  difference  between  this  screen  and  the  summary  screen? 
What  the  difference? 
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7.  Did  your  plan  do  well  on  the  areas  that  are  important  to  you?  Was 
this  an  important  factor  in  your  final  decision  to  choose  a  plan? 

8 .  How  did  you  decide  what  areas  are  important  to  you? 

9.  How  valuable  is  the  total  stars  category  to  you?  (Do  you  people  click 
on  this  section) 

10.  How  valuable  is  the  "Did  this  plan  do  well..."  information  to  you.  (Are 
people  using  this  section?) 

11. In  choosing  your  plan  did  you  weight  the  areas  depending  on  how 
relevant  they  were  to  your  life  or  did  you  simply  choose  the  plan 
with  the  highest  number  of  stars? 

12. Was  it  easy  to  find  this  screen  while  navigating  around  the  site? 


Other  Survey  Results 

(Are  people  going  to  this  screen?  Especially  people  who  have  children  or 

people  who  use  a  lot  of  care.) 

1.  Do  you  think  that  you  would  only  use  this  information  if  you  have  a 
child  or  someone  who  consistently  needs  a  lot  of  care? 

2.  Do  you  think  that  the  information  given  on  the  summary  and  details 
screens  of  the  child  care  information  and  the  people  who  have  used  a 
lot  of  care  information  is  as  useful  as  the  information  on  the 
general  survey? 

3.  Would  you  be  less  likely  to  use  these  survey  results  after  reviewing 
the  original  survey  summary,  details  and  results? 
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4 .  Do  you  think  that  these  survey  topics  are  the  most  useful  to  learn 
about  the  benefits  of  each  health  care  plan  to  children  and  people 
who  use  a  lot  of  care? 

5.  Is  it  easy  to  find  this  screen  while  navigating  around  the  site? 


Your  Worksheet 

1.  How  likely  are  you  to  print  this  screen  out  and  use  it? 

2.  Would  you  find  the  site  just  as  useful  without  this  screen?  Why  or 
why  not? 

3 .  Did  you  know  that  by  clicking  on  the  underlined  words  you  can  find 
out  more  about  the  particular  topics?  Is  this  fact  made  clear  by  the 
format? 

4.  With  this  sheet,  do  you  feel  like  you  have  enough  information  to  make 
an  informed  decision  about  the  best  health  care  provider  for  you  and 
your  family? 

5.  Did  you  click  on  the  question  marks  to  help  you  match  the  plans  to 
your  needs? 

6.  Does  whether  the  plan  is  a  PPO  or  HMO  matter  in  your  decision-making? 
Does  the  site  give  you  more  importance  on  how  your  plan' s  status  as 
an  HMO  or  PPO  effect  the  benefits  of  each  health  care  plan? 

7.  Is  there  more  information  that  you  need  to  make  a  decision  after 
seeing  your  results  on  this  screen?  What?  What  can  be  eliminated? 

8 .  Was  it  easy  to  find  this  screen  while  navigating  around  the  site? 

Did  you  click  on  the  doctors,  costs  and  benefits  information? 
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Getting  more  information.  Contact  information 

1.  Does  this  screen  provide  you  with  useful  information? 

2.  Is  it  new  information  or  things  you  already  knew  about? 

3.  How  likely  would  you  be  to  use  this  to  fill  in  your  worksheet 

4.  Do  the  places  to  call  and  visit  before  deciding  on  a  health  plan  seem 
reasonable  and  useful? 

(How  important  is  it  for  you  to  know  the  opinions  that  other  people  have 
about  the  health  plans  that  you  can  choose  from)  ? 

Okay,  now  I'm  going  to  have  you  fill  out  this  questionnaire  which  asks 
you  a  little  more  about  the  site  and  also  for  some  basic  information 
about  yourself.  Do  you  have  any  questions  or  comments? 

Mark  End  Time 
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APPENDIX  2:  ROUND  2  PROTOCOL 


Subject  #  _ 

Start  time _ 

End  time  _ 

Computer  Station 


Task  1:  Free  exploration  of  Decision  Helper.  Interviewer  is 
unobtrusively  taking  notes  and  is  available  to  answer  questions.  (30 
minutes) . 

Task  2:  Unstructured  interview:  Participant  gives  overall  impressions 
on  usability  and  usefulness  of  site.  (20  minutes). 

Task  3:  Structured  interview.  Interviewer  and  participant  go  through 
site  screen  by  screen.  (40  minutes) . 

Task  4:  Written  questionnaire:  Demographic  and  website  usability  survey 
(approximately  20  minutes) 

INTERVIEW  INTRODUCTION 
I.  Website  purpose 

a.  Decision  Helper  Tool-used  to  help  people  select  a  health 
plan 

b.  To  be  distributed  and  customized  by  employer  (FFS) 

II.  Interview  Structure 

a.  Interview  purpose 

■  To  evaluate  website,  not  participant's  abilities 

■  Confidentiality 

■  To  make  sure  that  participant  knows  how  to  use 
computer  features  (this  should  have  been  screened 
for  at  recruitment) 

b.  During  free  exploration  section  encourage  participant  to 
give  spontaneous  thoughts  and  impressions  of  screens 

c.  Assess  key  demographic  characteristics 

■  Does  participant  have  children  covered  by  health 

plan? _ 
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■  Does  participant  have  anyone  who  needs  a  lot  of  care 
covered  by  his  or  her  personal  health  plan? 


FREE  EXPLORATION 

Take  as  long  as  you  want  to  go  through  the  site.  You  don't  have  to  go 

through  in  any  particular  order.  Just  use  your  judgment  and  let  me  know 

why  you  skip  around  and  what  type  of  order  you  are  going  in. 

Are  you  ready  to  begin?  Okay  you'll  start  on  this  first  screen,  which 

is  already  loaded. 

Mark  Begin  Time _ 

Interviewer  should  unobtrusively  observe  and  take  notes  on  subject's 
reactions  to  website. 

Note: 

I)  Time  spent  on: 

■  Introductory  screens 

■  Types  of  plans  information  (do  they  seem  confused  by  HMO/PPO 
information) 

Whether  people  use  child  care/People  who  use  lots  of  care 

II)  Search  Pattern  Sequence 

■  Screens  missed 

■  Whether  all  bar  screens  are  reviewed 

■  Navigation  techniques/problems 

■  Use  of  Worksheet  screen 

■  Visibility  of  on-screen  links 


UNSTRUCTURED  INTERVIEW 

Gather  opinions  on: 

■  Overall  ease  of  use  (usability) 

■  Usefulness  of  information 

■  Best/worst  aspects  of  Decision  Helper 
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■  Clarity  of  site  navigation  and  language 

■  Perceived  importance  of  survey  information 

■  Necessity  of  instructions  for  use 

■  Impressions  on  appropriate  website  audience 

■  Site  structure  (Survey  Intro,  Your  Worksheet  placement) 

■  After  participants  have  viewed  the  site  can  they 
choose/eliminate  any  plans? 

STRUCTURED  INTERVIEW 

Now,  we're  going  to  go  through  each  of  the  screens  of  the  website.  I 
want  to  walk  through  the  site  going  from  screen  to  screen  in  the  order 
that  you  went  through  it.  On  each  screen,  I'd  like  you  to  mention 
anything  you  found  especially  confusing  or  especially  useful.  I'd  also 
like  you  to  tell  me  what  you  think  the  purpose  of  each  screen  is,  or 
what  information  is  being  conveyed.  (At  the  end  of  each  section  ask  if 
there  is  anything  else  they  would  like  to  add  or  any  questions  they  may 
have . ) 

(Note  what  navigation  tools  participant  uses  to  move  from  one  screen  to 
the  next . ) 

See  if  people  click  to  expand  on  the  major  topics  to  see  subtopics  or 
just  move  on  to  next  major  heading.  Check  if  this  is  a  reason  that  they 
are  skipping  screens.) 

Questions 

For  all  screens  that  participants  missed  go  through  how  they  could  have 
gotten  there  and  whether  this  was  a  screen  that  they  would  have  found 
useful . 

A)  Splash  Screen 

1.  Overall  impressions  of  screen:  Opinions  on  "A  Guide  to  Health 
Plan  Quality  from  the  Consumer  Point  of  View"  title.  Did 
participant's  understand  meaning? 

2.  Impressions  of  picture  subjects  (do  participant's  identify  with 
this  picture) .  Why  or  why  not  would  this  screen  draw 
participants  in  to  look  at  the  rest  of  the  site. 

B)  Welcome  Section 
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■  Do  interviewees  feel  that  their  health  care  experiences  will 
be  similar  to  survey  participants 

E)  Survey  Topics  Screen 

1.  Topics  chosen 

■  Most  important  in  choosing  a  plan  (what  to  add  take  out) 

■  Applicability  to  participant's  health  plan  decisions 

■  When  going  through  topics,  are  they  weighted  differently 

2.  Usability 

■  Ability  to  understand  what  defined  each  topic 

■  Category  breadth  (too  broad,  not  broad  enough) 

(Note  whether  participant  goes  to  summary  or  details  screen  first) 

F)  Summary  Charts 

Note:  At  what  point,  if  any,  do  participants  click  on  the  "what  do 

the  stars  mean  screen"? 

How  much  time  is  spent  on  this  screen  relative  to  details  screen? 

1 .  Usability 

■  Likelihood  of  reading  screen  instructions 

■  Star  instructions  (Do  people  think  stars  represent  comparison 
.  to  survey  average  or  simple  plan  ratings-ask  participant  to 

give  you  star  meanings?) 

■  Is  "what  do  the  stars  tell  you"  necessary  for  understanding 
star  chart? 

■  Are  participants  comparing  across  all  plans  or  HMO  vs.  PPO 
plans? 

■  Ease  of  navigation  to  summary  descriptor  screen  (noticing  on¬ 
screen  links) 

2 .  Usefulness 

■  Can  health  plan  choice  be  made  on  this  information  alone 

■  After  viewing  summary  tables  are  details  tables  necessary 

■  Should  infoiunation  be  more  specific  to  individual  groups 
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G)  Details  screen  (bar  graphs) 


Note:  At  what  point,  if  any,  do  participants  click  on  the  "what  do 
the  bars  mean  screen" 

How  much  time  to  participants  spend  looking  at  what  do  bar  graphs 
tell  you  screen?  (Are  they  noticing  this  link?) 

Time  spent  on  this  screen  vs.  Star  chart  screens 

1.  Usability 

■  Topic  definition  clarity 

■  What  topics  are  being  graphed  (bullet  points  or  more  general 
topics ) 

■  Is  it  clear  that  the  bar  graphs  charting  one  topic  each  (not 
all  7  in  one  graph) 

■  Understanding  of  bar  graph  colors 

■  Is  bar  graph  explanation  screen  necessary  (can  participants 
find  it) 

■  Usability  in  relation  to  summary  screens 

■  Do  participants  realize  that  star  and  bar  graphs  charting  the 
same  survey  information 

■  Ease  of  screen  navigation  (ability  to  go  back  and  forth 
between  this  a  summary  screen) 

■  Clarity  of  language 

2.  Usefulness 

■  Topics  to  be  added 

■  Could  participant  choose /eliminate  a  health  plan  by  these 
charts  alone  (what  more  information  is  needed) 

■  With  these  charts  are  summary  screens  necessary 

■  Should  topics  be  more  specific  to  individual  groups 
(children,  elderly,  etc.) 

H)  Review  the  Results 
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1.  Usability 

■  Apparent  difference  between  this  and  suiranary  screen 

■  Are  screen  instructions  visible  are  they  used 

■  Is  it  apparent  that  this  screen  links  to  worksheet  screen 

■  Are  people  filling  out  "this  plan  did  well  on  topics 
important  to  me" 

■  "Click  here"-is  it  used 

2.  Usefulness 

■  "Click  here"  does  it  cause  people  to  not  look  at  specific 
topics,  but  only  look  at  the  averages 

■  What  value  does  it  add  to  the  site 

■  Useful  to  have  information  connect  to  Your  Worksheet  screen 

■  Effectiveness  in  aiding  health  care  decision 

■  How  do  participant's  decide  which  topics  to  put  more  weight 
on 

I)  Other  Survey  Results 

(Are  people  going  to  this  screen?  Especially  people  who  have  children  or 
people  who  use  a  lot  of  care?) 

1.  Usability 

■  People  viewing  on-screen  links 

■  How  are  they  navigating  to  results  screens 

■  Are  these  results  from  the  same  survey  as  general  survey 
results-do  participants  understand  survey  results 

2.  Usefulness 

■  Used  only  by  people  with  kids/those  who  use  lots  of  care  or 
all  participants 

■  Value  of  summary/details  screens  in  comparison  to  "general 
survey  results" 

■  Most  useful  topics  to  learn  about  benefits  for  these  specific 
groups 
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J)  Your  Worksheet 

1.  Usability 

■  Visibility  of  on-screen  links 

■  People  using  the  screen  correctly  (are  they  trying  to  type 
information  onto  the  screen) 

■  Is  this  screen  at  the  right  part  in  the  site's  structure 

■  Do  people  see  that  clicking  on  question  marks  will  lead  them 
back  to  Review  the  Results  screen 

■  Are  people  viewing  screen  instructions 

■  What  instructional  information  needed  to  make  screen  use  more 
clear 

■  Ease  of  navigation  to  Your  Worksheet  and  back  to  other 
screens 

■  Likeliness  of  using  Your  Worksheet 

2.  Usefulness 

■  Does  this  screen  add  value  to  the  site? 

■  With  information  on  this  screen,  can  people  decide  upon  or 
against  a  health  plan? 

■  Does  HMO/PPO  status  of  a  plan  effect  whether  it  will  be  used? 

■  Is  more  information  needed  to  make  plan  useful?  What  can  be 
eliminated? 

K)  Getting  more  information.  Contact  information 

1.  Usability 

■  Is  it  clear  what  information  will  be  provided  by  title 

*  Is  this  information  understandable 

■  Likeliness  of  following  recommendations  (reasonable 
requests?) 

■  Perceived  validity  of  information  listed  on  these  screens 


INTRODUCE  WRITTEN  QUESTIONNAIRE.  SOLICIT  QUESTIONS  OR  ADDITIONAL 
COMMENTS  ON  SITE 


Mark  End  Time 
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WRITTEN  QUESTIONNAIRE 


Section  A:  Usability 

1.  Imagine  that  you  must  choose  a  health  plan  right  now.  Based  on  the 
health  plan  information  you  looked  at  today,  which  health  plan  would 
you  be  most  likely  to  enroll  in? 

1  □  Coastal  Health  Plan 

2  □  Pacific  Health  Plan 

3  □  Atlantic  Health  Plan 

4  □  Mountain  Health  Plan 

2.  Why  would  you  be  most  likely  to  choose  this  plan?  What  did  you  like 
most  about  this  plan?  (check  all  that  apply) 

1  □  Overall,  the  plan  was  rated  highly  by  the  Decision  Helper 
survey 

2  □  The  plan  was  rated  highly  on  a  specific  survey  topic 

3  □  Other _ 

3.  Did  you  use  what  people  said  about  plans  in  the  Decision  Helper  to 
help  you  decide  which  plan  to  choose? 

1  □  Yes 

2  □  No 

4.  How  hard  or  easy  was  it  for  you  to  choose  a  health  plan  with  the 
infoimiation  you  were  given  today? 

1  □  Very  hard 

2  □  Somewhat  hard 

3  □  Somewhat  easy 

4  □  Very  easy 
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5.  With  only  the  health  plan  information  on  the  Decision  Helper,  how 
sure  are  you  that  you  could  choose  the  plan  that  is  best  for  you? 


1 

□ 

Not  at  all  sure 

2 

□ 

Slightly  sure 

3 

□ 

Somewhat  sure 

4 

□ 

Very  sure 

6.  Please  rate  the  usefulness  of  the  following  sections  of  the  Decision 
Helper. 


D" 

Welcome"  section 

1 

□ 

Not  at  all  useful 

2 

□ 

Slightly  useful 

3 

□ 

Somewhat  useful 

4 

□ 

Very  useful 

2) "Types  of  Plans"  section  (containing  information  on  HMOs 


and 

PPOs) 

1 

□ 

Not  at  all  useful 

2 

□ 

Slightly  useful 

3 

□ 

Somewhat  useful 

4 

□ 

Very  useful 

4) "Survey  Results"  section  overall 


1 

□ 

Not  at  all  useful 

2 

□ 

Slightly  useful 

3 

□ 

Somewhat  useful 

4 

□ 

Very  useful 
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5)  "Suinmary"  sub-section  (containing  star  chart) 


1 

□ 

Not  at  all  useful 

2 

□ 

Slightly  useful 

3 

□ 

Somewhat  useful 

4 

□ 

Very  useful 

6) "Details"  sub-section  (containing  bar  graphs) 


1 

□ 

Not  at  all  useful 

2 

□ 

Slightly  useful 

3 

□ 

Somewhat  useful 

4 

□ 

Very  useful 

7) "Review  the  Results"  sub-section 

1  □  Not  at  all  useful 

2  □  Slightly  useful 

3  □  Somewhat  useful 

4  □  Very  useful 


8 ) "Children' s  Care"  sub-section 


1 

□ 

Not  at  all  useful 

2 

□ 

Slightly  useful 

3 

□ 

Somewhat  useful 

4 

□ 

Very  useful 
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9) "People  Who  Have  Used  a  Lot  of  Care"  sub-section 


1  □  Not  at  all  useful 

2  □  Slightly  useful 

3  □  Somewhat  useful 

4  □  Very  useful 

10) "Your  Worksheet"  section 

1  □  Not  at  all  useful 

2  □  Slightly  useful 

3  □  Somewhat  useful 

4  □  Very  useful 

7.  Which  is  more  useful  (when  comparing  quality  of  health  plans)? 

1  □  "Summary"  sub-section  (containing  star  chart) 

2  □  "Details"  sub-section  (containing  bars  graphs) 


8 .  If  you  could  have  only  the  summary  sub-section  or  the  details  sub¬ 
section,  which  would  you  prefer  to  have  when  choosing  a  health 
plan 

a.  □  Summary  siib-section  (containing  star  charts) 

b.  □  Details  sub-section  (containing  bars  charts) 

9.  Who  do  you  think  created  the  Decision  Helper? 


1 

□ 

Health  plans 

2 

□ 

A  government  agency 

3 

□ 

An  independent  research  organization 

4 

□ 

Other  ( 

5 

□ 

Don' t  know 
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10.  Who  do  you  think  paid  for  the  survey  described  in  Decision  Helper? 

1  □  Health  plans 

2  □  A  government  agency 

3  □  An  independent  research  organization 

4  □  Other  (  _ ) 

5  □  Don't  know 

11.  Do  you  think  the  survey  results  described  in  the  Decision  Helper  tell 
you  what  the  health  plans  are  really  like? 

1  □  Yes 

2  □  No 

3  □  Not  sure/don't  know 

12.  How  easy  or  hard  was  it  for  you  to  find  what  you  were  looking  for  on 
this  website? 

1  □  Very  Easy 

2  □  Somewhat  easy 

3  □  Somewhat  hard 

4  □  Very  hard 

13. Overall,  how  easy  or  hard  was  it  for  you  to  understand  the 
information  on  the  Decision  Helper 

1  □  Very  Easy  To  Understand 

2  □  Somewhat  Easy  To  Understand 

3  □  Somewhat  Hard  To  Understand 

4  □  Very  Hard  To  Understand 

14. Overall,  how  useful  do  you  think  that  the  Decision  Helper  would  be 
for 

1)  Choosing  a  new  health  plan 

1  □  Very  Useful 

2  □  Somewhat  useful 

3  □  Not  at  all  useful 
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2)  Switching  health  plans 

1  □  Very  Useful 

2  □  Somewhat  useful 

3  □  Not  at  all  useful 

3)  Comparing  your  health  plan  to  other  plans 

1  □  Very  Useful 

2  □  Somewhat  useful 

3  □  Not  at  all  useful 


15. Did  the  Decision  Helper  cover  the  information  that  you  expected  it  to 
cover? 
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16.  Look  at  the  amount  of  information  about  "Types  of  Plans".  Which 
statement  below  best  matches  how  you  feel  about  the  amount  of 
information  you  would  have  liked  before  deciding  which  plans  to 
choose? 

1  □  I  would  have  liked  a  lot  more  information  about  types  of 

plans . 

2^1  would  have  liked  a  little  more  information  about  types  of 
plans . 

3  □  I  had  about  the  right  amount  of  information  about  types  of 

plans . 

4  □  I  would  have  liked  a  little  less  information  about  types  of 

plans . 

5  □  I  would  have  liked  a  lot  less  infoomiation  about  types  of 

plans . 

17.  Look  at  the  amount  of  information  about  "Survey  Results".  Which 
statement  below  best  matches  how  you  feel  about  the  amount  of 
information  you  would  have  liked  before  deciding  which  plans  to 
choose? 

1  □  I  would  have  liked  a  lot  more  information  about  survey 

results . 

2  0  1  would  have  liked  a  little  more  information  about 

survey  results. 

3  □  I  had  about  the  right  amount  of  information  about 

survey  results. 

4^1  would  have  liked  a  little  less  information  about 
survey  results. 

5  □  I  would  have  liked  a  lot  less  information  about  survey 

results . 
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18. Consider  the  information  that  was  not  in  the  Decision  Helper  that  you 
would  have  liked.  Which  statement  below  best  matches  how  you  feel 
about  the  amount  of  other  information  you  would  have  liked  before 
deciding  which  plans  to  choose? 

1  □  I  would  have  liked  a  lot  more  other  information. 

2  □  I  would  have  liked  a  little  more  other  information. 

3  □  I  had  about  the  right  amount  of  other  information. 

4  □  I  would  have  liked  a  little  less  about  other  information. 

5  □  I  would  have  liked  a  lot  less  about  other  information. 


19. What  other  information  would  you  have  liked  (circle  all  that  apply) 

1  □  Services  Covered 

2  □  Costs  per  plan 

3  □  Doctors  in  plan  networks 

4  □  Quality  of  Doctors 

5  □  Other _ 

20. In  general,  how  much  time  do  you  spend  when  making  decision? 

1  □  Very  long  time 

2  □  Somewhat  long  time 

3  □  Somewhat  short  time 

4  □  Very  short  time 


21.  In  general,  how  much  time  do  you  spend  when  selecting  a  health  plan? 

1  □  Very  long  time 

2  □  Somewhat  long  time 

3  □  Somewhat  short  time 

4  □  Very  short  time 
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DEMOGRAPHIC  QUESTIONNAIRE  (FFS) 

1 .  What  is  your  age  now? 


1  □  18  to  24 

2  D  25to34 

30  35  to  44 

4  □  45  to  54 

5  □  55  to  64 

6  □  65  or  older 


2 .  Are  you : 

1  □  Female 

2  □  Male 


3.  How  would  you  describe  your  race  or  ethnicity? 


1  □  American  Indian  or  Alaskan  Native 

2  □  Asian  or  Pacific  Islander 

3  □  Black  or  African  American 

4  □  Hispanic  or  Latino 

5  □  White 

6  □  Another  race/ethnicity  or  multicultural  (write  in) 

4.  What  is  the  highest  grade  or  level  of  education  that  you  have 
completed? 


1  □  8^^  grade  or  less 

2  □  Some  high  school 

3  □  High  school  graduate  or  GED 

4  □  1-3  years  of  college  (including  associates  degree) 

5  □  4  year  college  degree  or  more 
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5.  In  general 


how  would  you  rate  your  overall  health? 


1  □  Excellent 

2  □  Very  good 

3  □  Good 

4  □  Fair 

5  □  Poor 

6.  Do  you  have  a  child  who  is  covered  by  your  personal  health  plan? 

1  □  Yes 

2  0  No 

7.  Is  there  another  adult  who  is  covered  by  your  personal  health  plan? 

1  □  Yes 

2  □  No 

8  .  Do  you  have  someone  who  needs  a  lot  of  care  covered  by  your  personal 
health  plan? 

1  □  Yes 

2  □  No 

9,  What  was  your  combined  family  income  last  year  (1998-1999) 

1  □  $0  to  $10,000 

2  n  $10,001  to  $14,999 

3  □  $15,000  to  $24,999 

4  □  $25,000  to  $34,999 

5  □  $35,000  to  $49,999 

6  □  $50,000  to  $74,999 

7  □  $75, 000“$100, 000 

8  □  over  $100,000 
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10. What  is  the  name  of  your  current  health  plan? 


11.  How  many  months  or  years  in  a  row  have  you  been  covered  by  the  same 
health  insurance? 

1  □  3  months  or  less 

2  □  4  to  6  months 

3  □  7  to  9  months 

4  □  10  to  11  months 

5  □  1  to  2  years 

6  □  2  to  4  years 

7  □  more  than  4  years 

12.  Please  rate  your  overall  level  of  satisfaction  with  your  health 
plan. 

1  □  Very  dissatisfied 

2  □  Somewhat  dissatisfied 

3  □  Somewhat  satisfied 

4  □  Very  satisfied 


13. How  much  time  do  you  spend  on  a  computer  during  the  average  week? 


AT  HOME 

1  □  None  at  all 

2  n  0-5  hours 

3  □  6-10  hours 

4  □  11-20  hours 

5  □  21-30  hours 

6  □  30+  hours 


AT  WORK 

1  □  None  at  all 

2  □  0-5  hours 

3  □  6-10  hours 

4  □  11-20  hours 

5  □  21-30  hours 

6  □  30+  hours 


OTHER  {EX.  LIBRARY) 
iD  None  at  all 
2n  0-5  hours 
3^  6-10  hours 
4^  11-20  hours 
sD  21-30  hours 
eD  30+  hours 


79 


APPENDIX  3:  DECISION  HELPER,  VERSION  2.0  SCREENS-ROUND  1  OF  USABILITY  TESTING 


Figure  3A:  "Splash"  Screen.  Round 


Weico 


Figure  3B:  **Welcome”  Screen.  Round  1. 


Survey  Results 


Summary  screen"  with  Star  Charts.  Round 


Survey  Results 


Figure  3D:  "Details"  Screen  with  Bar  Graphs.  Round 


Figure  3E:  "Review  the  Results"  Screen.  Round  1. 


Completinq  Your  Worksheet 


3F:  "Your  Worksheet"  Screen.  Round 


APPENDIX  4:  DECISION  HELPER,  VERSION  2.0  SCREENS-ROUND  2  OF  USABILITY  TESTING. 


Figure  4A:  "Splash"  Screen.  Round  1. 


Figure  4B:  "Welcome"  Screen.  Round  2. 


Figure  4C:  '^Summary''  Screen  with  Start  Charts.  Round  2. 
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Figure  4D:  "Details"  Screen  with  Bar  Graphs.  Round  2. 


3ins1rudtons  -  Microsolt  internet  Explorer  provi...  BsW! 


4E:  "Review  the  Results"  Screen  with  Informational  Pop-Up  Screen.  Round  2. 
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APPENDIX  5:  ROUND  2.  ALTERNATE  SPLASH  SCREENS. 
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